Oct 30 11:53:28.021370 ip-10-0-131-76 systemd[1]: Starting Kubernetes Kubelet... Oct 30 11:53:28.423669 ip-10-0-131-76 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:53:28.423669 ip-10-0-131-76 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 11:53:28.423669 ip-10-0-131-76 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:53:28.423669 ip-10-0-131-76 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:53:28.427910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427826 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:28.427910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427910 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:28.427910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427913 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427917 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427919 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427922 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427926 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427929 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427932 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427935 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427938 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427941 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427944 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427951 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427954 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427957 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427959 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427962 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427965 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427967 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427970 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427973 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:28.428002 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427975 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:28.428517 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427978 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:28.428517 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427980 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:28.428517 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427983 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:28.428517 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427986 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:28.428517 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427988 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:28.428517 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427991 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:28.428517 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427994 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:28.428517 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427996 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:28.428517 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.427999 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:28.428517 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428002 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:28.428517 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428005 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:28.428517 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428008 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:28.428517 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428010 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:28.428517 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428014 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:28.428517 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428017 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:28.428517 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428021 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:28.428517 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428026 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:28.428517 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428029 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:28.428517 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428031 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:28.428963 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428034 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:28.428963 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428037 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:28.428963 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428040 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:28.428963 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428043 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:28.428963 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428046 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:28.428963 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428048 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:28.428963 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428051 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:28.428963 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428054 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:28.428963 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428056 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:28.428963 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428059 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:28.428963 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428061 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:28.428963 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428064 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:28.428963 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428078 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:28.428963 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428081 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:28.428963 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428085 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:28.428963 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428088 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:28.428963 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428091 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:28.428963 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428096 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:28.428963 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428099 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:28.429427 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428102 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:28.429427 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428105 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:28.429427 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428108 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:28.429427 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428112 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:28.429427 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428115 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:28.429427 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428118 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:28.429427 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428120 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:28.429427 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428123 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:28.429427 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428126 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:28.429427 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428129 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:28.429427 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428132 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:28.429427 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428134 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:28.429427 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428137 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:28.429427 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428140 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:28.429427 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428143 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:28.429427 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428146 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:28.429427 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428149 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:28.429427 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428152 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:28.429427 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428154 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:28.429910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428157 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:28.429910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428159 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:28.429910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428162 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:28.429910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428164 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:28.429910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428167 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:28.429910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428169 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:28.429910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428173 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:28.429910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428176 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:28.429910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428179 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:28.429910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428182 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:28.429910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428184 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:28.429910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428187 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:28.429910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428189 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:28.429910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428192 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:28.429910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428195 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:28.429910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428197 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:28.429910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428200 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:28.429910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428203 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:28.429910 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428205 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428208 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428211 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428213 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428216 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428218 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428221 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.428225 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429044 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429051 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429056 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429060 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429063 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429080 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429083 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429087 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429092 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429095 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429098 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429102 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:28.430362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429105 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:28.430839 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429109 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:28.430839 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429112 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:28.430839 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429115 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:28.430839 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429118 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:28.430839 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429121 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:28.430839 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429124 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:28.430839 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429127 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:28.430839 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429129 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:28.430839 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429132 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:28.430839 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429135 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:28.430839 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429137 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:28.430839 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429140 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:28.430839 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429142 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:28.430839 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429145 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:28.430839 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429148 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:28.430839 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429150 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:28.430839 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429153 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:28.430839 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429156 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:28.430839 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429159 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429161 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429164 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429166 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429169 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429172 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429175 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429178 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429181 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429183 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429186 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429188 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429191 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429194 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429197 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429199 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429202 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429205 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429208 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429210 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:28.431298 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429213 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:28.431774 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429216 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:28.431774 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429218 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:28.431774 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429221 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:28.431774 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429223 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:28.431774 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429226 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:28.431774 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429228 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:28.431774 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429231 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:28.431774 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429233 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:28.431774 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429236 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:28.431774 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429238 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:28.431774 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429242 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:28.431774 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429244 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:28.431774 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429247 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:28.431774 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429249 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:28.431774 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429252 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:28.431774 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429255 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:28.431774 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429257 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:28.431774 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429266 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:28.431774 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429269 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429272 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429275 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429277 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429280 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429282 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429285 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429287 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429290 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429293 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429295 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429298 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429300 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429303 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429306 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429308 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429310 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429313 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429315 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429318 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:28.432279 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429321 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:28.432759 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429323 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:28.432759 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429326 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:28.432759 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429328 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:28.432759 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429331 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:28.432759 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429334 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:28.432759 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429338 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:28.432759 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429340 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:28.432759 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429343 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:28.432759 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429345 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:28.432759 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429348 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:28.432759 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429350 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:28.432759 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429358 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:28.432759 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429361 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:28.432759 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429381 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:28.432759 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.429385 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:28.432759 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430708 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:28.432759 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430716 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:28.432759 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430720 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:28.432759 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430723 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:28.433222 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430726 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:28.433222 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430729 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:28.433222 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430732 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:28.433222 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430734 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:28.433222 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430737 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:28.433222 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430740 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:28.433222 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430743 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:28.433222 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430745 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:28.433222 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430748 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:28.433222 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430751 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:28.433222 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430754 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:28.433222 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430757 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:28.433222 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430759 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:28.433222 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430762 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:28.433222 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430765 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:28.433222 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430768 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:28.433222 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430771 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:28.433222 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430773 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:28.433222 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430776 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:28.433663 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430778 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:28.433663 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430781 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:28.433663 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430784 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:28.433663 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430786 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:28.433663 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430789 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:28.433663 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430792 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:28.433663 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430795 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:28.433663 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430797 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:28.433663 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430800 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:28.433663 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430803 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:28.433663 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430805 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:28.433663 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430808 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:28.433663 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430811 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:28.433663 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430813 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:28.433663 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430816 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:28.433663 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430819 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:28.433663 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430822 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:28.433663 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430825 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:28.433663 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430828 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430831 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430833 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430836 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430839 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430841 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430844 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430847 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430850 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430853 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430856 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430858 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430861 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430864 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430866 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430869 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430871 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430875 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430879 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430882 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:28.434125 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430888 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430891 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430894 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430896 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430899 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430901 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430904 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430906 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430909 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430912 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430914 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430917 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430919 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430922 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430924 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430927 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430930 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430932 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430935 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430937 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:28.434678 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430940 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:28.435167 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430942 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:28.435167 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430945 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:28.435167 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430947 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:28.435167 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430950 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:28.435167 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430952 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:28.435167 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430955 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:28.435167 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430957 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:28.435167 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430960 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:28.435167 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430962 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:28.435167 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430964 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:28.435167 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430969 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:28.435167 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430973 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:28.435167 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430976 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:28.435167 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430979 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:28.435167 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430982 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:28.435167 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430984 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:28.435167 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430987 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:28.435167 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430990 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:28.435167 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430994 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430996 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.430999 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.431002 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.431004 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.431006 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432784 2574 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432800 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432807 2574 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432812 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432817 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432821 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432826 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432831 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432835 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432839 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432842 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432846 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432849 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432852 2574 flags.go:64] FLAG: --cgroup-root="" Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432855 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432859 2574 flags.go:64] FLAG: --client-ca-file="" Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432862 2574 flags.go:64] FLAG: --cloud-provider="external" Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432865 2574 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 11:53:28.435610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432870 2574 flags.go:64] FLAG: --cluster-domain="" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432873 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432877 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432887 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432891 2574 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432895 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432898 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432902 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432905 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432908 2574 flags.go:64] FLAG: --contention-profiling="false" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432912 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432915 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432918 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432921 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432926 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432929 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432932 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432935 2574 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432938 2574 flags.go:64] FLAG: --enable-server="true" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432944 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432948 2574 flags.go:64] FLAG: --event-burst="100" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432952 2574 flags.go:64] FLAG: --event-qps="50" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432955 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432958 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 11:53:28.436168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432961 2574 flags.go:64] FLAG: --eviction-hard="" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432966 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432969 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432972 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432976 2574 flags.go:64] FLAG: --eviction-soft="" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432979 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432986 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432989 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432992 2574 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432995 2574 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.432998 2574 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433001 2574 flags.go:64] FLAG: --feature-gates="" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433011 2574 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433014 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433017 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433021 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433024 2574 flags.go:64] FLAG: --healthz-port="10248" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433027 2574 flags.go:64] FLAG: --help="false" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433030 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-131-76.ec2.internal" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433033 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433036 2574 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433039 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433042 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 11:53:28.436803 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433046 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433049 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433052 2574 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433055 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433060 2574 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433063 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433076 2574 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433079 2574 flags.go:64] FLAG: --kube-reserved="" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433082 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433085 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433088 2574 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433091 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433094 2574 flags.go:64] FLAG: --lock-file="" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433096 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433099 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433103 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433108 2574 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433112 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433115 2574 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433118 2574 flags.go:64] FLAG: --logging-format="text" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433121 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433124 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433133 2574 flags.go:64] FLAG: --manifest-url="" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433136 2574 flags.go:64] FLAG: --manifest-url-header="" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433141 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 11:53:28.437355 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433144 2574 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433148 2574 flags.go:64] FLAG: --max-pods="110" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433151 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433154 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433157 2574 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433160 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433163 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433166 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433169 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433176 2574 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433180 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433184 2574 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433187 2574 flags.go:64] FLAG: --pod-cidr="" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433190 2574 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433193 2574 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433196 2574 flags.go:64] FLAG: --pods-per-core="0" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433199 2574 flags.go:64] FLAG: --port="10250" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433208 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433211 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0541f4526f2866583" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433215 2574 flags.go:64] FLAG: --qos-reserved="" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433217 2574 flags.go:64] FLAG: --read-only-port="10255" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433221 2574 flags.go:64] FLAG: --register-node="true" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433224 2574 flags.go:64] FLAG: --register-with-taints="" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433228 2574 flags.go:64] FLAG: --registry-burst="10" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433231 2574 flags.go:64] FLAG: --registry-qps="5" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433234 2574 flags.go:64] FLAG: --reserved-cpus="" Oct 30 11:53:28.437943 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433237 2574 flags.go:64] FLAG: --reserved-memory="" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433241 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433244 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433247 2574 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433252 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433255 2574 flags.go:64] FLAG: --runonce="false" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433258 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433262 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433265 2574 flags.go:64] FLAG: --seccomp-default="false" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433268 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433270 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433273 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433277 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433280 2574 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433283 2574 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433286 2574 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433289 2574 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433293 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433296 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433299 2574 flags.go:64] FLAG: --system-cgroups="" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433302 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433308 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433311 2574 flags.go:64] FLAG: --tls-cert-file="" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433314 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433318 2574 flags.go:64] FLAG: --tls-min-version="" Oct 30 11:53:28.438567 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433321 2574 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433324 2574 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433327 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433330 2574 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433333 2574 flags.go:64] FLAG: --v="2" Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433338 2574 flags.go:64] FLAG: --version="false" Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433342 2574 flags.go:64] FLAG: --vmodule="" Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433346 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433350 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433489 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433493 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433499 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433502 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433505 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433508 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433511 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433513 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433516 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433518 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433521 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433524 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433526 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:28.439162 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433529 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:28.439697 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433533 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:28.439697 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433535 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:28.439697 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433538 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:28.439697 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433540 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:28.439697 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433543 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:28.439697 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433545 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:28.439697 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433548 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:28.439697 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433550 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:28.439697 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433553 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:28.439697 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433556 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:28.439697 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433558 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:28.439697 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433561 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:28.439697 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433563 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:28.439697 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433566 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:28.439697 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433568 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:28.439697 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433571 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:28.439697 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433573 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:28.439697 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433578 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:28.439697 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433581 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433584 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433588 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433591 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433595 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433597 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433600 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433602 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433605 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433608 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433610 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433613 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433615 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433618 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433620 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433623 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433625 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433628 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433630 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433633 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:28.440155 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433635 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:28.440628 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433638 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:28.440628 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433640 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:28.440628 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433643 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:28.440628 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433645 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:28.440628 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433648 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:28.440628 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433651 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:28.440628 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433653 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:28.440628 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433655 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:28.440628 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433658 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:28.440628 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433661 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:28.440628 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433664 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:28.440628 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433667 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:28.440628 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433669 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:28.440628 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433673 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:28.440628 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433676 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:28.440628 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433679 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:28.440628 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433682 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:28.440628 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433686 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:28.440628 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433690 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433693 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433696 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433699 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433702 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433704 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433707 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433710 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433713 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433716 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433718 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433721 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433723 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433726 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433729 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433731 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433734 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433736 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433739 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433741 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:28.441130 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433744 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:28.441595 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433746 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:28.441595 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433749 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:28.441595 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433753 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:28.441595 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433755 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:28.441595 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433758 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:28.441595 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433760 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:28.441595 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433764 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:28.441595 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433767 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:28.441595 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433770 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:28.441595 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433772 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:28.441595 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433775 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:28.441595 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433778 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:28.441595 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433781 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:28.441595 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433783 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:28.441595 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.433811 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:53:28.441595 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433848 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433852 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433855 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433858 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433861 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433863 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433866 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433869 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433871 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433874 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433876 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433879 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433882 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433884 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433887 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433890 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433892 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433910 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433915 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433920 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:28.441981 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433924 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433927 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433929 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433932 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433937 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433939 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433942 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433945 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433948 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433951 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433953 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433956 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433958 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433961 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433963 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433966 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433969 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433971 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433974 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433976 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:28.442467 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433979 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:28.442926 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433981 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:28.442926 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433984 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:28.442926 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433986 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:28.442926 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433989 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:28.442926 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433992 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:28.442926 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433994 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:28.442926 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433996 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:28.442926 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.433999 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:28.442926 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434001 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:28.442926 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434005 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:28.442926 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434010 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:28.442926 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434013 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:28.442926 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434016 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:28.442926 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434018 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:28.442926 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434021 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:28.442926 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434025 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:28.442926 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434028 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:28.442926 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434031 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:28.442926 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434033 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434036 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434039 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434041 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434044 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434046 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434049 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434052 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434054 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434057 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434059 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434062 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434065 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434080 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434082 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434085 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434088 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434091 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434094 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434096 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:28.443443 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434099 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:28.443909 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434101 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:28.443909 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434104 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:28.443909 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434107 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:28.443909 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434111 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:28.443909 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434113 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:28.443909 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434116 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:28.443909 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434118 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:28.443909 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434121 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:28.443909 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434125 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:28.443909 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434131 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:28.443909 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434135 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:28.443909 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434138 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:28.443909 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434142 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:28.443909 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434145 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:28.443909 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434149 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:28.443909 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434151 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:28.443909 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434154 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:28.443909 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434157 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:28.443909 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434160 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:28.444362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434162 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:28.444362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434165 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:28.444362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434168 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:28.444362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434170 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:28.444362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434173 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:28.444362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434176 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:28.444362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.434179 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:28.444362 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.441223 2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 11:53:28.444362 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.441239 2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 11:53:28.444362 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.441257 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 11:53:28.444362 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.441262 2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 11:53:28.444362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441320 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:28.444362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441325 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:28.444362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441329 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:28.444362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441332 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:28.444362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441335 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:28.444362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441338 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:28.444362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441341 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:28.444362 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441343 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441347 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441349 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441352 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441355 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441358 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441360 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441363 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441366 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441368 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441371 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441373 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441376 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441379 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441381 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441384 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441386 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441389 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441391 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441394 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:28.444824 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441397 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:28.445296 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441399 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:28.445296 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441404 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:28.445296 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441407 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:28.445296 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441410 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:28.445296 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441413 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:28.445296 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441416 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:28.445296 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441419 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:28.445296 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441421 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:28.445296 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441424 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:28.445296 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441427 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:28.445296 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441430 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:28.445296 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441433 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:28.445296 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441435 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:28.445296 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441438 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:28.445296 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441440 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:28.445296 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441443 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:28.445296 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441446 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:28.445296 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441448 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:28.445296 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441451 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441454 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441457 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441459 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441462 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441464 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441467 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441470 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441472 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441475 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441477 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441480 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441483 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441486 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441488 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441492 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441495 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441498 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441500 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441503 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:28.445735 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441506 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:28.446213 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441508 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:28.446213 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441511 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:28.446213 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441514 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:28.446213 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441516 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:28.446213 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441519 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:28.446213 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441522 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:28.446213 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441524 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:28.446213 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441527 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:28.446213 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441529 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:28.446213 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441532 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:28.446213 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441534 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:28.446213 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441537 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:28.446213 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441540 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:28.446213 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441544 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:28.446213 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441547 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:28.446213 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441550 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:28.446213 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441552 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:28.446213 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441555 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:28.446213 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441558 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441560 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441563 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441566 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441568 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441571 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441573 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441576 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441579 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441582 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441585 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441588 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441591 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441593 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441596 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441599 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441602 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441606 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441609 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441613 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:28.446686 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441615 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:28.447164 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.441643 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:53:28.447164 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441785 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:28.447164 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441789 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:28.447164 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441792 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:28.447164 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441795 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:28.447164 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441798 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:28.447164 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441800 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:28.447164 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441803 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:28.447164 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441806 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:28.447164 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441808 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:28.447164 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441811 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:28.447164 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441814 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:28.447164 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441816 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:28.447164 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441819 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:28.447164 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441821 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:28.447164 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441824 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:28.447537 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441826 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:28.447537 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441829 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:28.447537 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441832 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:28.447537 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441835 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:28.447537 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441837 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:28.447537 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441840 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:28.447537 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441843 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:28.447537 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441846 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:28.447537 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441849 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:28.447537 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441851 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:28.447537 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441854 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:28.447537 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441857 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:28.447537 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441859 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:28.447537 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441862 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:28.447537 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441865 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:28.447537 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441867 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:28.447537 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441870 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:28.447537 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441873 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:28.447537 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441878 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441881 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441884 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441886 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441889 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441891 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441894 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441896 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441899 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441901 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441904 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441907 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441909 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441912 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441915 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441917 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441920 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441923 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441925 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441928 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:28.447966 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441931 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:28.448451 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441934 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:28.448451 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441936 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:28.448451 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441939 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:28.448451 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441941 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:28.448451 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441944 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:28.448451 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441946 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:28.448451 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441949 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:28.448451 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441951 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:28.448451 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441954 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:28.448451 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441956 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:28.448451 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441959 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:28.448451 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441961 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:28.448451 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441964 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:28.448451 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441966 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:28.448451 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441969 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:28.448451 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441971 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:28.448451 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441974 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:28.448451 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441976 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:28.448451 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441979 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441982 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441985 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441987 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441990 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441993 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441996 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.441999 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442001 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442004 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442007 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442010 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442014 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442017 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442020 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442023 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442025 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442028 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442030 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442033 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:28.448901 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442035 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:28.449371 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442038 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:28.449371 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442040 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:28.449371 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442043 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:28.449371 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442045 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:28.449371 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442048 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:28.449371 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442050 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:28.449371 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442053 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:28.449371 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442055 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:28.449371 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442058 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:28.449371 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442060 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:28.449371 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442063 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:28.449371 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442082 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:28.449371 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442086 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:28.449371 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.442110 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:53:28.449371 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442141 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:28.449371 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442145 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442148 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442152 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442155 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442158 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442161 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442163 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442166 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442169 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442171 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442174 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442178 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442182 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442185 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442188 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442190 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442193 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442197 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442199 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442202 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:28.449747 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442205 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442208 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442210 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442213 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442216 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442218 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442221 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442223 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442226 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442229 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442231 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442234 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442236 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442239 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442242 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442244 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442247 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442249 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442252 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442254 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:28.450229 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442257 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:28.450685 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442259 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:28.450685 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442262 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:28.450685 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442265 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:28.450685 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442267 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:28.450685 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442276 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:28.450685 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442278 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:28.450685 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442281 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:28.450685 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442284 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:28.450685 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442286 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:28.450685 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442289 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:28.450685 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442291 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:28.450685 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442294 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:28.450685 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442297 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:28.450685 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442299 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:28.450685 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442302 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:28.450685 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442304 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:28.450685 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442307 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:28.450685 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442310 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:28.450685 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442312 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:28.451258 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442315 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:28.451258 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442318 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:28.451258 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442320 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:28.451258 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442323 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:28.451258 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442325 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:28.451258 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442329 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:28.451258 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442333 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:28.451258 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442337 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:28.451258 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442339 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:28.451258 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442342 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:28.451258 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442345 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:28.451258 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442347 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:28.451258 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442350 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:28.451258 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442352 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:28.451258 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442355 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:28.451258 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442358 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:28.451258 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442361 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:28.451258 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442363 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:28.451258 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442366 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442369 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442372 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442374 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442377 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442379 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442382 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442384 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442387 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442389 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442392 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442394 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442397 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442399 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442402 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442404 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442407 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442409 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442412 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442415 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:28.451717 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442418 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:28.452534 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442420 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:28.452534 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442423 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:28.452534 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442425 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:28.452534 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442428 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:28.452534 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442430 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:28.452534 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:28.442435 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:28.452534 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.442491 2574 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 11:53:28.452534 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.444775 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 11:53:28.452534 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.445732 2574 server.go:1014] "Starting client certificate rotation" Oct 30 11:53:28.452534 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.445829 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 11:53:28.452534 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.446792 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 11:53:28.452534 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.448119 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 11:53:28.454207 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.454182 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 11:53:28.462961 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.462946 2574 log.go:25] "Validated CRI v1 runtime API" Oct 30 11:53:28.467769 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.467753 2574 log.go:25] "Validated CRI v1 image API" Oct 30 11:53:28.469169 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.469141 2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 11:53:28.472273 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.472252 2574 fs.go:135] Filesystem UUIDs: map[088d176a-8d8a-4d6d-9485-cb6c6e501c0c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 dcb41eed-532c-473d-9714-19b7d1a2818b:/dev/nvme0n1p3] Oct 30 11:53:28.472341 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.472273 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 11:53:28.477668 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.477557 2574 manager.go:217] Machine: {Timestamp:2026-10-30 11:53:28.476266025 +0000 UTC m=+0.336485010 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099529 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d37fbb3ae1420407e84be2d27641d SystemUUID:ec2d37fb-b3ae-1420-407e-84be2d27641d BootID:938eaf26-5289-4369-a053-15d2ca83361a Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d0:e5:f2:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d0:e5:f2:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:d9:ea:e9:f1:7d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 11:53:28.478292 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.478281 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 11:53:28.478380 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.478368 2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 11:53:28.481053 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.481024 2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 11:53:28.481215 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.481052 2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-76.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 11:53:28.481266 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.481217 2574 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 11:53:28.481266 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.481225 2574 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 11:53:28.481266 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.481251 2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 11:53:28.481266 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.481265 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 11:53:28.482417 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.482408 2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 11:53:28.486297 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.486284 2574 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 11:53:28.486426 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.486417 2574 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 11:53:28.487046 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.487036 2574 kubelet.go:493] "Attempting to sync node with API server" Oct 30 11:53:28.487095 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.487052 2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 11:53:28.487095 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.487063 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 11:53:28.487095 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.487086 2574 kubelet.go:405] "Adding apiserver pod source" Oct 30 11:53:28.487200 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.487100 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 11:53:28.487515 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.487502 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 11:53:28.489665 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.489648 2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 11:53:28.494455 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.494442 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 11:53:28.495266 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.495254 2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 11:53:28.495314 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.495296 2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 11:53:28.495453 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.495443 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 11:53:28.495495 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.495458 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 11:53:28.495495 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.495465 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 11:53:28.495495 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.495471 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 11:53:28.495495 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.495477 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 11:53:28.495495 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.495482 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 11:53:28.496064 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.496055 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 11:53:28.496109 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.496081 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 11:53:28.496109 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.496089 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 11:53:28.496109 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.496095 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 11:53:28.496187 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.496112 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 11:53:28.496187 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.496121 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 11:53:28.496763 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.496753 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 11:53:28.496794 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.496766 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 11:53:28.500659 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.500646 2574 server.go:1263] "Started kubelet" Oct 30 11:53:28.505559 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.500729 2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 11:53:28.505752 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.505593 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 11:53:28.505800 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.505787 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 11:53:28.505781 ip-10-0-131-76 systemd[1]: Started Kubernetes Kubelet. Oct 30 11:53:28.507087 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.507056 2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 11:53:28.509731 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.509719 2574 server.go:317] "Adding debug handlers to kubelet server" Oct 30 11:53:28.511992 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.511974 2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 11:53:28.513571 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.513552 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 11:53:28.514119 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.514104 2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 11:53:28.514208 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.514183 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-76.ec2.internal\" not found" Oct 30 11:53:28.514632 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.514614 2574 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 11:53:28.514632 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.514632 2574 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 11:53:28.514779 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.514616 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 11:53:28.514779 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.514753 2574 reconstruct.go:97] "Volume reconstruction finished" Oct 30 11:53:28.514779 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.514763 2574 reconciler.go:29] "Reconciler: start to sync state" Oct 30 11:53:28.515171 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.515158 2574 factory.go:55] Registering systemd factory Oct 30 11:53:28.515233 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.515194 2574 factory.go:223] Registration of the systemd container factory successfully Oct 30 11:53:28.515413 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.515398 2574 factory.go:153] Registering CRI-O factory Oct 30 11:53:28.515450 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.515418 2574 factory.go:223] Registration of the crio container factory successfully Oct 30 11:53:28.515477 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.515471 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 11:53:28.515505 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.515491 2574 factory.go:103] Registering Raw factory Oct 30 11:53:28.515539 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.515507 2574 manager.go:1199] Started watching for new ooms in manager Oct 30 11:53:28.516445 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.516430 2574 manager.go:319] Starting recovery of all containers Oct 30 11:53:28.525918 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.525900 2574 manager.go:324] Recovery completed Oct 30 11:53:28.532798 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.532781 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:28.534678 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.534659 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:28.534744 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.534698 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:28.534744 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.534715 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:28.535689 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.535656 2574 cpu_manager.go:226] "Starting" policy="none" Oct 30 11:53:28.535689 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.535684 2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 11:53:28.535836 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.535691 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:28.535836 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.535706 2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 11:53:28.536589 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.536572 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:28.536683 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.536594 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:28.536683 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.536604 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:28.537060 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.537038 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-76.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 11:53:28.537165 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.537093 2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-76.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 11:53:28.537949 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.536969 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-76.ec2.internal.18e34c847aa4d884 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-76.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-76.ec2.internal,},FirstTimestamp:2026-10-30 11:53:28.500607108 +0000 UTC m=+0.360826094,LastTimestamp:2026-10-30 11:53:28.500607108 +0000 UTC m=+0.360826094,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-76.ec2.internal,}" Oct 30 11:53:28.538038 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.538009 2574 policy_none.go:50] "Start" Oct 30 11:53:28.538038 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.538025 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 11:53:28.538164 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.538040 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 11:53:28.540128 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.540114 2574 policy_none.go:44] "Start" Oct 30 11:53:28.549308 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.549288 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 11:53:28.557362 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.557266 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-76.ec2.internal.18e34c847cacbd3e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-76.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-76.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-76.ec2.internal,},FirstTimestamp:2026-10-30 11:53:28.534678846 +0000 UTC m=+0.394897833,LastTimestamp:2026-10-30 11:53:28.534678846 +0000 UTC m=+0.394897833,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-76.ec2.internal,}" Oct 30 11:53:28.557362 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.557343 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-76.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 11:53:28.557362 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.557343 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 11:53:28.557516 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.557343 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 11:53:28.590421 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.576389 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 11:53:28.590421 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.577543 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 11:53:28.590421 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.577565 2574 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 11:53:28.590421 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.577583 2574 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 11:53:28.590421 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.577614 2574 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 11:53:28.590421 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.580946 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-85lxw" Oct 30 11:53:28.590421 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.581701 2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 11:53:28.590421 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.586406 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-76.ec2.internal.18e34c847cad2954 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-76.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-76.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-76.ec2.internal,},FirstTimestamp:2026-10-30 11:53:28.534706516 +0000 UTC m=+0.394925712,LastTimestamp:2026-10-30 11:53:28.534706516 +0000 UTC m=+0.394925712,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-76.ec2.internal,}" Oct 30 11:53:28.590421 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.587050 2574 manager.go:341] "Starting Device Plugin manager" Oct 30 11:53:28.590421 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.587600 2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 11:53:28.590421 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.587613 2574 server.go:87] "Starting device plugin registration server" Oct 30 11:53:28.590421 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.587820 2574 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 11:53:28.590421 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.587829 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 11:53:28.590421 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.588148 2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 11:53:28.590421 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.588238 2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 11:53:28.590421 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.588247 2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 11:53:28.590421 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.588843 2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 11:53:28.590421 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.588890 2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-76.ec2.internal\" not found" Oct 30 11:53:28.592791 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.592773 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-85lxw" Oct 30 11:53:28.603655 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.603582 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-76.ec2.internal.18e34c847cad628e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-76.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-76.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-76.ec2.internal,},FirstTimestamp:2026-10-30 11:53:28.534721166 +0000 UTC m=+0.394940152,LastTimestamp:2026-10-30 11:53:28.534721166 +0000 UTC m=+0.394940152,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-76.ec2.internal,}" Oct 30 11:53:28.632898 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.632871 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:53:28.678568 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.678476 2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal"] Oct 30 11:53:28.678568 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.678566 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:28.679579 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.679562 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:28.679695 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.679595 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:28.679695 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.679609 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:28.681544 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.681528 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:28.681712 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.681697 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" Oct 30 11:53:28.681759 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.681729 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:28.682473 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.682457 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:28.682544 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.682488 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:28.682544 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.682501 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:28.682544 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.682463 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:28.682639 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.682562 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:28.682639 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.682575 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:28.684320 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.684301 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal" Oct 30 11:53:28.684410 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.684329 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:28.685786 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.685768 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:28.685867 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.685794 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:28.685867 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.685803 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:28.688464 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.688449 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:28.689153 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.689138 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:28.689234 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.689166 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:28.689234 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.689180 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:28.689234 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.689212 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-76.ec2.internal" Oct 30 11:53:28.708765 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.708607 2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-76.ec2.internal" Oct 30 11:53:28.708854 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.708772 2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-76.ec2.internal\": node \"ip-10-0-131-76.ec2.internal\" not found" Oct 30 11:53:28.715329 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.715311 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac067dd06bb744f9ffd35d4622ecd99a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal\" (UID: \"ac067dd06bb744f9ffd35d4622ecd99a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" Oct 30 11:53:28.715380 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.715337 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac067dd06bb744f9ffd35d4622ecd99a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal\" (UID: \"ac067dd06bb744f9ffd35d4622ecd99a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" Oct 30 11:53:28.715380 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.715355 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a981c242c496637c2b85dbce301f463-config\") pod \"kube-apiserver-proxy-ip-10-0-131-76.ec2.internal\" (UID: \"5a981c242c496637c2b85dbce301f463\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal" Oct 30 11:53:28.731440 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.731421 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-76.ec2.internal\" not found" node="ip-10-0-131-76.ec2.internal" Oct 30 11:53:28.734251 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.734235 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-76.ec2.internal\" not found" node="ip-10-0-131-76.ec2.internal" Oct 30 11:53:28.751653 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.751635 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-76.ec2.internal\" not found" Oct 30 11:53:28.815568 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.815537 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a981c242c496637c2b85dbce301f463-config\") pod \"kube-apiserver-proxy-ip-10-0-131-76.ec2.internal\" (UID: \"5a981c242c496637c2b85dbce301f463\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal" Oct 30 11:53:28.815686 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.815575 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac067dd06bb744f9ffd35d4622ecd99a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal\" (UID: \"ac067dd06bb744f9ffd35d4622ecd99a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" Oct 30 11:53:28.815686 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.815596 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a981c242c496637c2b85dbce301f463-config\") pod \"kube-apiserver-proxy-ip-10-0-131-76.ec2.internal\" (UID: \"5a981c242c496637c2b85dbce301f463\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal" Oct 30 11:53:28.815686 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.815601 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac067dd06bb744f9ffd35d4622ecd99a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal\" (UID: \"ac067dd06bb744f9ffd35d4622ecd99a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" Oct 30 11:53:28.815686 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.815658 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac067dd06bb744f9ffd35d4622ecd99a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal\" (UID: \"ac067dd06bb744f9ffd35d4622ecd99a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" Oct 30 11:53:28.815686 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:28.815658 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac067dd06bb744f9ffd35d4622ecd99a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal\" (UID: \"ac067dd06bb744f9ffd35d4622ecd99a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" Oct 30 11:53:28.852062 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.852036 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-76.ec2.internal\" not found" Oct 30 11:53:28.952867 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:28.952781 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-76.ec2.internal\" not found" Oct 30 11:53:29.034332 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:29.034294 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" Oct 30 11:53:29.037298 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:29.037278 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal" Oct 30 11:53:29.053590 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:29.053539 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-76.ec2.internal\" not found" Oct 30 11:53:29.154299 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:29.154256 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-76.ec2.internal\" not found" Oct 30 11:53:29.254871 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:29.254794 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-76.ec2.internal\" not found" Oct 30 11:53:29.355286 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:29.355257 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-76.ec2.internal\" not found" Oct 30 11:53:29.445836 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:29.445805 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 11:53:29.446343 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:29.445974 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 11:53:29.456140 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:29.456111 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-76.ec2.internal\" not found" Oct 30 11:53:29.512198 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:29.512128 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:53:29.538883 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:29.538854 2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:53:29.556361 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:29.556329 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-76.ec2.internal\" not found" Oct 30 11:53:29.578081 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:29.578033 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac067dd06bb744f9ffd35d4622ecd99a.slice/crio-4028eaff961969401c3554706f9bf3f13c181ffa0560bcbc20040c0aef366984 WatchSource:0}: Error finding container 4028eaff961969401c3554706f9bf3f13c181ffa0560bcbc20040c0aef366984: Status 404 returned error can't find the container with id 4028eaff961969401c3554706f9bf3f13c181ffa0560bcbc20040c0aef366984 Oct 30 11:53:29.578323 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:29.578303 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a981c242c496637c2b85dbce301f463.slice/crio-eee0a121e54d3c931a8ffcd5feed923fac046341f5f69d4af853f871c712bd1a WatchSource:0}: Error finding container eee0a121e54d3c931a8ffcd5feed923fac046341f5f69d4af853f871c712bd1a: Status 404 returned error can't find the container with id eee0a121e54d3c931a8ffcd5feed923fac046341f5f69d4af853f871c712bd1a Oct 30 11:53:29.582955 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:29.582941 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 11:53:29.597521 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:29.597486 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 11:48:29 +0000 UTC" deadline="2028-07-16 03:37:21.78944134 +0000 UTC" Oct 30 11:53:29.597521 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:29.597519 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14991h43m52.191925042s" Oct 30 11:53:29.657349 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:29.657318 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-76.ec2.internal\" not found" Oct 30 11:53:29.749888 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:29.749863 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:29.751196 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:29.751176 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:29.751262 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:29.751210 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:29.751262 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:29.751224 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:29.758436 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:29.758418 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-76.ec2.internal\" not found" Oct 30 11:53:29.858554 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:29.858479 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-76.ec2.internal\" not found" Oct 30 11:53:29.923125 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:29.923095 2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:53:30.014732 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.014692 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" Oct 30 11:53:30.040381 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.040242 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:53:30.041485 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.041463 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal" Oct 30 11:53:30.061496 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.061469 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:53:30.371029 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.370992 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:53:30.487907 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.487875 2574 apiserver.go:52] "Watching apiserver" Oct 30 11:53:30.514665 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.514629 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 11:53:30.518387 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.518362 2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 11:53:30.518710 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.518683 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-htqhw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6","openshift-multus/multus-additional-cni-plugins-d7bvp","kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal","openshift-multus/multus-6pxtt","openshift-ovn-kubernetes/ovnkube-node-qq48w","kube-system/global-pull-secret-syncer-mqv9x","kube-system/konnectivity-agent-qjvts","openshift-dns/node-resolver-nhbdl","openshift-image-registry/node-ca-7c297","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal","openshift-cluster-node-tuning-operator/tuned-2jgjj","openshift-multus/network-metrics-daemon-rh7rw","openshift-network-diagnostics/network-check-target-ld8pr"] Oct 30 11:53:30.521447 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.521424 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7c297" Oct 30 11:53:30.523842 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.523819 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:30.523942 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:30.523894 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh7rw" podUID="260d2659-42a8-461f-8994-0c5f9522a80d" Oct 30 11:53:30.525791 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.525768 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.527898 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.527879 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:30.527990 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:30.527948 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ld8pr" podUID="363a0adc-193f-48d3-9628-1b9d10665597" Oct 30 11:53:30.530225 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.530205 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.534436 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.533531 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.536284 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.536258 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nhbdl" Oct 30 11:53:30.537461 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.537441 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 11:53:30.537553 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.537519 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 11:53:30.537553 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.537529 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 11:53:30.537659 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.537624 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pxw6w\"" Oct 30 11:53:30.539198 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.539177 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:30.539298 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:30.539237 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqv9x" podUID="46289f03-406c-4831-9ad1-0e6751fd7bd5" Oct 30 11:53:30.541333 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.541317 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.543811 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.543783 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-htqhw" Oct 30 11:53:30.544731 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.544710 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 11:53:30.544823 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.544738 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 11:53:30.544823 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.544792 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 11:53:30.544937 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.544856 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 11:53:30.544937 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.544910 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jxkrq\"" Oct 30 11:53:30.545013 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.544975 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 11:53:30.551805 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.551781 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qjvts" Oct 30 11:53:30.554454 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.554435 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.578427 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.578401 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 11:53:30.578541 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.578473 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7b4f8" Oct 30 11:53:30.583691 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.583641 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" event={"ID":"ac067dd06bb744f9ffd35d4622ecd99a","Type":"ContainerStarted","Data":"4028eaff961969401c3554706f9bf3f13c181ffa0560bcbc20040c0aef366984"} Oct 30 11:53:30.586513 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.586479 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal" event={"ID":"5a981c242c496637c2b85dbce301f463","Type":"ContainerStarted","Data":"eee0a121e54d3c931a8ffcd5feed923fac046341f5f69d4af853f871c712bd1a"} Oct 30 11:53:30.616347 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.616312 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 11:53:30.626026 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.625950 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj598\" (UniqueName: \"kubernetes.io/projected/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-kube-api-access-hj598\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.626026 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.625993 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-socket-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.626281 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626053 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-hostroot\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.626281 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626103 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-multus-conf-dir\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.626281 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626152 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-run-ovn\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.626281 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626184 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e-host\") pod \"node-ca-7c297\" (UID: \"29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e\") " pod="openshift-image-registry/node-ca-7c297" Oct 30 11:53:30.626281 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626219 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs\") pod \"network-metrics-daemon-rh7rw\" (UID: \"260d2659-42a8-461f-8994-0c5f9522a80d\") " pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:30.626281 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626246 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp7n7\" (UniqueName: \"kubernetes.io/projected/82a77175-bb28-474f-b4ec-dac73d790410-kube-api-access-sp7n7\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.626568 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626289 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-run\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.626568 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626315 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-tmp\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.626568 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626357 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-device-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.626568 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626405 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-sys-fs\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.626568 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626449 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx8v8\" (UniqueName: \"kubernetes.io/projected/b2937779-b03d-4a5b-87bc-f062283eeb2e-kube-api-access-bx8v8\") pod \"iptables-alerter-htqhw\" (UID: \"b2937779-b03d-4a5b-87bc-f062283eeb2e\") " pod="openshift-network-operator/iptables-alerter-htqhw" Oct 30 11:53:30.626568 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626486 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d897418e-880a-46a9-8b13-9c2887bfddb7-system-cni-dir\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.626568 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626511 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d897418e-880a-46a9-8b13-9c2887bfddb7-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.626891 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626658 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jzbb\" (UniqueName: \"kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb\") pod \"network-check-target-ld8pr\" (UID: \"363a0adc-193f-48d3-9628-1b9d10665597\") " pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:30.626891 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626684 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-multus-cni-dir\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.626891 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626712 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/554f0b9b-bae8-478b-b3b1-85b376a79cc9-env-overrides\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.626891 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626736 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-etc-kubernetes\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.626891 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626768 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f15ba445-e38a-46f6-be0f-65f7307e79d6-konnectivity-ca\") pod \"konnectivity-agent-qjvts\" (UID: \"f15ba445-e38a-46f6-be0f-65f7307e79d6\") " pod="kube-system/konnectivity-agent-qjvts" Oct 30 11:53:30.626891 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626792 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-run-k8s-cni-cncf-io\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.626891 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626813 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-run-netns\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.626891 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626836 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/617427c7-590d-4d80-8c4e-e11c7812dca0-tmp-dir\") pod \"node-resolver-nhbdl\" (UID: \"617427c7-590d-4d80-8c4e-e11c7812dca0\") " pod="openshift-dns/node-resolver-nhbdl" Oct 30 11:53:30.626891 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626869 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:30.627334 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626892 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-sysctl-d\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.627334 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626914 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-systemd\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.627334 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626939 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d897418e-880a-46a9-8b13-9c2887bfddb7-cnibin\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.627334 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626963 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5g55\" (UniqueName: \"kubernetes.io/projected/d897418e-880a-46a9-8b13-9c2887bfddb7-kube-api-access-s5g55\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.627334 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.626997 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-var-lib-openvswitch\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.627334 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627024 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-sys\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.627334 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627045 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-host\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.627334 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627083 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-tuned\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.627334 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627110 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d897418e-880a-46a9-8b13-9c2887bfddb7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.627334 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627136 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f15ba445-e38a-46f6-be0f-65f7307e79d6-agent-certs\") pod \"konnectivity-agent-qjvts\" (UID: \"f15ba445-e38a-46f6-be0f-65f7307e79d6\") " pod="kube-system/konnectivity-agent-qjvts" Oct 30 11:53:30.627334 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627160 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-var-lib-cni-multus\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.627334 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627183 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/31df51a8-4f22-4745-b25f-8b69654f2845-multus-daemon-config\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.627334 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627207 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-slash\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.627334 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627230 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-lib-modules\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.627334 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627252 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-etc-selinux\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.627334 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627276 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-log-socket\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.628089 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627311 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-cni-bin\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.628089 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627334 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/554f0b9b-bae8-478b-b3b1-85b376a79cc9-ovn-node-metrics-cert\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.628089 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627392 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/554f0b9b-bae8-478b-b3b1-85b376a79cc9-ovnkube-config\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.628089 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627418 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46289f03-406c-4831-9ad1-0e6751fd7bd5-dbus\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:30.628089 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627446 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-kubernetes\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.628089 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627470 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-cnibin\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.628089 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627498 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-var-lib-cni-bin\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.628089 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627524 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8tg9\" (UniqueName: \"kubernetes.io/projected/31df51a8-4f22-4745-b25f-8b69654f2845-kube-api-access-v8tg9\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.628089 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627549 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-run-openvswitch\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.628089 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627584 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jtv5\" (UniqueName: \"kubernetes.io/projected/554f0b9b-bae8-478b-b3b1-85b376a79cc9-kube-api-access-7jtv5\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.628089 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627609 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-run-ovn-kubernetes\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.628089 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627633 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d897418e-880a-46a9-8b13-9c2887bfddb7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.628089 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627658 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b2937779-b03d-4a5b-87bc-f062283eeb2e-iptables-alerter-script\") pod \"iptables-alerter-htqhw\" (UID: \"b2937779-b03d-4a5b-87bc-f062283eeb2e\") " pod="openshift-network-operator/iptables-alerter-htqhw" Oct 30 11:53:30.628089 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627693 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-multus-socket-dir-parent\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.628089 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627716 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-cni-netd\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.628089 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627744 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/617427c7-590d-4d80-8c4e-e11c7812dca0-hosts-file\") pod \"node-resolver-nhbdl\" (UID: \"617427c7-590d-4d80-8c4e-e11c7812dca0\") " pod="openshift-dns/node-resolver-nhbdl" Oct 30 11:53:30.628832 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627767 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jhg2\" (UniqueName: \"kubernetes.io/projected/617427c7-590d-4d80-8c4e-e11c7812dca0-kube-api-access-5jhg2\") pod \"node-resolver-nhbdl\" (UID: \"617427c7-590d-4d80-8c4e-e11c7812dca0\") " pod="openshift-dns/node-resolver-nhbdl" Oct 30 11:53:30.628832 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627804 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-kubelet\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.628832 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627828 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-systemd-units\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.628832 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627854 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-var-lib-kubelet\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.628832 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627882 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/554f0b9b-bae8-478b-b3b1-85b376a79cc9-ovnkube-script-lib\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.628832 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627907 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-run-multus-certs\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.628832 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627932 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l95t8\" (UniqueName: \"kubernetes.io/projected/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e-kube-api-access-l95t8\") pod \"node-ca-7c297\" (UID: \"29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e\") " pod="openshift-image-registry/node-ca-7c297" Oct 30 11:53:30.628832 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627956 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2937779-b03d-4a5b-87bc-f062283eeb2e-host-slash\") pod \"iptables-alerter-htqhw\" (UID: \"b2937779-b03d-4a5b-87bc-f062283eeb2e\") " pod="openshift-network-operator/iptables-alerter-htqhw" Oct 30 11:53:30.628832 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.627981 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d897418e-880a-46a9-8b13-9c2887bfddb7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.628832 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.628006 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-system-cni-dir\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.628832 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.628029 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31df51a8-4f22-4745-b25f-8b69654f2845-cni-binary-copy\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.628832 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.628055 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.628832 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.628099 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-sysconfig\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.628832 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.628137 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e-serviceca\") pod \"node-ca-7c297\" (UID: \"29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e\") " pod="openshift-image-registry/node-ca-7c297" Oct 30 11:53:30.628832 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.628162 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8ccv\" (UniqueName: \"kubernetes.io/projected/260d2659-42a8-461f-8994-0c5f9522a80d-kube-api-access-h8ccv\") pod \"network-metrics-daemon-rh7rw\" (UID: \"260d2659-42a8-461f-8994-0c5f9522a80d\") " pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:30.628832 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.628187 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-run-systemd\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.629590 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.628210 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-sysctl-conf\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.629590 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.628235 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.629590 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.628258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-registration-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.629590 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.628284 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-etc-openvswitch\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.629590 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.628308 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-node-log\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.629590 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.628333 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-run-netns\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.629590 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.628357 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46289f03-406c-4831-9ad1-0e6751fd7bd5-kubelet-config\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:30.629590 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.628381 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-modprobe-d\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.629590 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.628406 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-var-lib-kubelet\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.638727 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.638698 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 11:53:30.638878 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.638850 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 11:53:30.639006 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.638980 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:53:30.639140 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.639025 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 11:53:30.639140 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.639032 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 11:53:30.639140 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.639061 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 11:53:30.639140 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.639118 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 11:53:30.639140 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.639132 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 11:53:30.639405 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.639025 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sw2hc\"" Oct 30 11:53:30.639405 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.639216 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 11:53:30.639405 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.639252 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 11:53:30.639405 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.639299 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tz47t\"" Oct 30 11:53:30.639405 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.639335 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:53:30.639405 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.639339 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 11:53:30.639405 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.639360 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-llkx2\"" Oct 30 11:53:30.639715 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.639583 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 11:53:30.639715 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.639650 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 11:53:30.639802 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.639724 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lrxbh\"" Oct 30 11:53:30.639851 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.639828 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 11:53:30.639973 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.639960 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6hs8x\"" Oct 30 11:53:30.640130 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.640115 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7cm7r\"" Oct 30 11:53:30.640205 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.640185 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 11:53:30.640205 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.640188 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 11:53:30.640284 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.640231 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8n4dr\"" Oct 30 11:53:30.640387 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.640368 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 11:53:30.640604 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.640585 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 11:53:30.729217 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729182 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-multus-socket-dir-parent\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.729411 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729221 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-cni-netd\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.729411 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729254 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/617427c7-590d-4d80-8c4e-e11c7812dca0-hosts-file\") pod \"node-resolver-nhbdl\" (UID: \"617427c7-590d-4d80-8c4e-e11c7812dca0\") " pod="openshift-dns/node-resolver-nhbdl" Oct 30 11:53:30.729411 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729304 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jhg2\" (UniqueName: \"kubernetes.io/projected/617427c7-590d-4d80-8c4e-e11c7812dca0-kube-api-access-5jhg2\") pod \"node-resolver-nhbdl\" (UID: \"617427c7-590d-4d80-8c4e-e11c7812dca0\") " pod="openshift-dns/node-resolver-nhbdl" Oct 30 11:53:30.729411 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729321 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-kubelet\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.729411 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-systemd-units\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.729411 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729361 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-var-lib-kubelet\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.729411 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729385 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/554f0b9b-bae8-478b-b3b1-85b376a79cc9-ovnkube-script-lib\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.729411 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729408 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-run-multus-certs\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.729737 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729431 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l95t8\" (UniqueName: \"kubernetes.io/projected/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e-kube-api-access-l95t8\") pod \"node-ca-7c297\" (UID: \"29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e\") " pod="openshift-image-registry/node-ca-7c297" Oct 30 11:53:30.729737 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729455 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2937779-b03d-4a5b-87bc-f062283eeb2e-host-slash\") pod \"iptables-alerter-htqhw\" (UID: \"b2937779-b03d-4a5b-87bc-f062283eeb2e\") " pod="openshift-network-operator/iptables-alerter-htqhw" Oct 30 11:53:30.729737 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729487 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d897418e-880a-46a9-8b13-9c2887bfddb7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.729737 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729517 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-system-cni-dir\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.729737 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729543 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31df51a8-4f22-4745-b25f-8b69654f2845-cni-binary-copy\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.729737 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729563 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.729737 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729579 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-sysconfig\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.729737 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729598 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e-serviceca\") pod \"node-ca-7c297\" (UID: \"29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e\") " pod="openshift-image-registry/node-ca-7c297" Oct 30 11:53:30.729737 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729617 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8ccv\" (UniqueName: \"kubernetes.io/projected/260d2659-42a8-461f-8994-0c5f9522a80d-kube-api-access-h8ccv\") pod \"network-metrics-daemon-rh7rw\" (UID: \"260d2659-42a8-461f-8994-0c5f9522a80d\") " pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:30.729737 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729634 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-run-systemd\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.729737 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729653 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-sysctl-conf\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.729737 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729667 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.729737 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729680 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-registration-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.729737 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729700 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-etc-openvswitch\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.729737 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729716 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-node-log\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.729737 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729729 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-run-netns\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.729737 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729743 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46289f03-406c-4831-9ad1-0e6751fd7bd5-kubelet-config\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:30.730278 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729760 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-modprobe-d\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.730278 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729788 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-var-lib-kubelet\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.730278 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729815 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj598\" (UniqueName: \"kubernetes.io/projected/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-kube-api-access-hj598\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.730278 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729841 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-socket-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.730278 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729856 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-hostroot\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.730278 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729869 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-multus-conf-dir\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.730278 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729891 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-run-ovn\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.730278 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729913 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e-host\") pod \"node-ca-7c297\" (UID: \"29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e\") " pod="openshift-image-registry/node-ca-7c297" Oct 30 11:53:30.730278 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729941 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs\") pod \"network-metrics-daemon-rh7rw\" (UID: \"260d2659-42a8-461f-8994-0c5f9522a80d\") " pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:30.730278 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729966 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp7n7\" (UniqueName: \"kubernetes.io/projected/82a77175-bb28-474f-b4ec-dac73d790410-kube-api-access-sp7n7\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.730278 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.729983 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-run\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.730278 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730014 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-tmp\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.730278 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730061 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-device-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.730278 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-sys-fs\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.730278 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730122 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx8v8\" (UniqueName: \"kubernetes.io/projected/b2937779-b03d-4a5b-87bc-f062283eeb2e-kube-api-access-bx8v8\") pod \"iptables-alerter-htqhw\" (UID: \"b2937779-b03d-4a5b-87bc-f062283eeb2e\") " pod="openshift-network-operator/iptables-alerter-htqhw" Oct 30 11:53:30.730278 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730148 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d897418e-880a-46a9-8b13-9c2887bfddb7-system-cni-dir\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.730278 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d897418e-880a-46a9-8b13-9c2887bfddb7-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.730858 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730201 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jzbb\" (UniqueName: \"kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb\") pod \"network-check-target-ld8pr\" (UID: \"363a0adc-193f-48d3-9628-1b9d10665597\") " pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:30.730858 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730253 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-multus-cni-dir\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.730858 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730274 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/554f0b9b-bae8-478b-b3b1-85b376a79cc9-env-overrides\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.730858 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730294 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-etc-kubernetes\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.730858 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730321 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f15ba445-e38a-46f6-be0f-65f7307e79d6-konnectivity-ca\") pod \"konnectivity-agent-qjvts\" (UID: \"f15ba445-e38a-46f6-be0f-65f7307e79d6\") " pod="kube-system/konnectivity-agent-qjvts" Oct 30 11:53:30.730858 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730351 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-run-k8s-cni-cncf-io\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.730858 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730374 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-run-netns\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.730858 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/617427c7-590d-4d80-8c4e-e11c7812dca0-tmp-dir\") pod \"node-resolver-nhbdl\" (UID: \"617427c7-590d-4d80-8c4e-e11c7812dca0\") " pod="openshift-dns/node-resolver-nhbdl" Oct 30 11:53:30.730858 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730433 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:30.730858 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730446 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-sysctl-d\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.730858 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730474 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-systemd\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.730858 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730521 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d897418e-880a-46a9-8b13-9c2887bfddb7-cnibin\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.730858 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730550 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5g55\" (UniqueName: \"kubernetes.io/projected/d897418e-880a-46a9-8b13-9c2887bfddb7-kube-api-access-s5g55\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.730858 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730574 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-var-lib-openvswitch\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.730858 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730600 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-sys\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.730858 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730624 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-host\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.730858 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730647 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-tuned\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.731465 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730670 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d897418e-880a-46a9-8b13-9c2887bfddb7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.731465 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730695 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f15ba445-e38a-46f6-be0f-65f7307e79d6-agent-certs\") pod \"konnectivity-agent-qjvts\" (UID: \"f15ba445-e38a-46f6-be0f-65f7307e79d6\") " pod="kube-system/konnectivity-agent-qjvts" Oct 30 11:53:30.731465 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730731 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-var-lib-cni-multus\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.731465 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730758 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/31df51a8-4f22-4745-b25f-8b69654f2845-multus-daemon-config\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.731465 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730794 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-slash\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.731465 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730817 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-lib-modules\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.731465 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730840 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-etc-selinux\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.731465 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730864 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-log-socket\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.731465 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730904 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-cni-bin\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.731465 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730930 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/554f0b9b-bae8-478b-b3b1-85b376a79cc9-ovn-node-metrics-cert\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.731465 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.730987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/554f0b9b-bae8-478b-b3b1-85b376a79cc9-ovnkube-config\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.731465 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.731019 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46289f03-406c-4831-9ad1-0e6751fd7bd5-dbus\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:30.731465 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.731040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-kubernetes\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.731465 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.731058 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-cnibin\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.731465 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.731097 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-var-lib-cni-bin\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.731465 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.731126 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8tg9\" (UniqueName: \"kubernetes.io/projected/31df51a8-4f22-4745-b25f-8b69654f2845-kube-api-access-v8tg9\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.731465 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.731157 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-run-openvswitch\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.732007 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.731178 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jtv5\" (UniqueName: \"kubernetes.io/projected/554f0b9b-bae8-478b-b3b1-85b376a79cc9-kube-api-access-7jtv5\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.732007 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.731202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-run-ovn-kubernetes\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.732007 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.731230 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d897418e-880a-46a9-8b13-9c2887bfddb7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.732007 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.731256 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b2937779-b03d-4a5b-87bc-f062283eeb2e-iptables-alerter-script\") pod \"iptables-alerter-htqhw\" (UID: \"b2937779-b03d-4a5b-87bc-f062283eeb2e\") " pod="openshift-network-operator/iptables-alerter-htqhw" Oct 30 11:53:30.732007 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.731960 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b2937779-b03d-4a5b-87bc-f062283eeb2e-iptables-alerter-script\") pod \"iptables-alerter-htqhw\" (UID: \"b2937779-b03d-4a5b-87bc-f062283eeb2e\") " pod="openshift-network-operator/iptables-alerter-htqhw" Oct 30 11:53:30.732187 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.732173 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-multus-socket-dir-parent\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.732247 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.732221 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-cni-netd\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.732370 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.732272 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/617427c7-590d-4d80-8c4e-e11c7812dca0-hosts-file\") pod \"node-resolver-nhbdl\" (UID: \"617427c7-590d-4d80-8c4e-e11c7812dca0\") " pod="openshift-dns/node-resolver-nhbdl" Oct 30 11:53:30.732612 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.732593 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-kubelet\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.732712 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.732645 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-systemd-units\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.732712 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.732696 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-var-lib-kubelet\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.733253 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.733236 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/554f0b9b-bae8-478b-b3b1-85b376a79cc9-ovnkube-script-lib\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.733343 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.733292 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-run-multus-certs\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.733543 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.733525 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2937779-b03d-4a5b-87bc-f062283eeb2e-host-slash\") pod \"iptables-alerter-htqhw\" (UID: \"b2937779-b03d-4a5b-87bc-f062283eeb2e\") " pod="openshift-network-operator/iptables-alerter-htqhw" Oct 30 11:53:30.734053 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.734030 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d897418e-880a-46a9-8b13-9c2887bfddb7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.734150 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.734128 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-system-cni-dir\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.734607 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.734592 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31df51a8-4f22-4745-b25f-8b69654f2845-cni-binary-copy\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.734682 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.734657 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.734734 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.734716 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-sysconfig\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.735090 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.735058 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e-serviceca\") pod \"node-ca-7c297\" (UID: \"29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e\") " pod="openshift-image-registry/node-ca-7c297" Oct 30 11:53:30.735351 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.735329 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-run-systemd\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.735484 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.735464 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-sysctl-conf\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.735536 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.735521 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.735591 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.735577 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-registration-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.735643 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.735620 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-etc-openvswitch\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.735693 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.735672 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-node-log\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.735739 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.735710 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-run-netns\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.735788 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.735754 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46289f03-406c-4831-9ad1-0e6751fd7bd5-kubelet-config\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:30.735862 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.735846 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-modprobe-d\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.735924 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.735906 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-var-lib-kubelet\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.736252 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.736234 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-socket-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.736343 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.736292 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-hostroot\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.736343 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.736328 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-multus-conf-dir\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.736445 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.736372 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-run-ovn\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.736445 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.736412 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e-host\") pod \"node-ca-7c297\" (UID: \"29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e\") " pod="openshift-image-registry/node-ca-7c297" Oct 30 11:53:30.736542 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:30.736507 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:30.736609 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:30.736577 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs podName:260d2659-42a8-461f-8994-0c5f9522a80d nodeName:}" failed. No retries permitted until 2026-10-30 11:53:31.236553273 +0000 UTC m=+3.096772248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs") pod "network-metrics-daemon-rh7rw" (UID: "260d2659-42a8-461f-8994-0c5f9522a80d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:30.736945 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.736923 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-run\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.737246 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.737221 2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 11:53:30.739364 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.739338 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-sys-fs\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.739447 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.739369 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46289f03-406c-4831-9ad1-0e6751fd7bd5-dbus\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:30.739447 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.739423 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-run-k8s-cni-cncf-io\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.739507 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.739459 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-run-ovn-kubernetes\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.739507 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.739492 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-run-openvswitch\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.739579 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.739560 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-cnibin\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.739650 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.739619 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d897418e-880a-46a9-8b13-9c2887bfddb7-system-cni-dir\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.739650 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.739621 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-var-lib-cni-bin\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.739707 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.739663 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-kubernetes\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.739707 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.739701 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/554f0b9b-bae8-478b-b3b1-85b376a79cc9-ovnkube-config\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.739764 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.739714 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-device-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.740932 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.740284 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-multus-cni-dir\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.741168 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.741137 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-etc-kubernetes\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.741591 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.741174 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d897418e-880a-46a9-8b13-9c2887bfddb7-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.741591 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.741532 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/554f0b9b-bae8-478b-b3b1-85b376a79cc9-env-overrides\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.748223 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.741957 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-etc-selinux\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.748223 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.742005 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-var-lib-cni-multus\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.748223 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.741996 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-log-socket\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.748223 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.742056 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-cni-bin\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.748223 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.742118 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-tuned\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.748223 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.742288 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-tmp\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.748223 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.742551 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/31df51a8-4f22-4745-b25f-8b69654f2845-multus-daemon-config\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.748223 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.742618 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-slash\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.748223 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.742711 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-lib-modules\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.748223 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.742778 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f15ba445-e38a-46f6-be0f-65f7307e79d6-konnectivity-ca\") pod \"konnectivity-agent-qjvts\" (UID: \"f15ba445-e38a-46f6-be0f-65f7307e79d6\") " pod="kube-system/konnectivity-agent-qjvts" Oct 30 11:53:30.748223 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.743272 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d897418e-880a-46a9-8b13-9c2887bfddb7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.748223 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.743431 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d897418e-880a-46a9-8b13-9c2887bfddb7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.748223 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.743485 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-systemd\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.748223 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.743523 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-run-netns\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.748223 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.743766 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/617427c7-590d-4d80-8c4e-e11c7812dca0-tmp-dir\") pod \"node-resolver-nhbdl\" (UID: \"617427c7-590d-4d80-8c4e-e11c7812dca0\") " pod="openshift-dns/node-resolver-nhbdl" Oct 30 11:53:30.748223 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:30.743846 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:30.748223 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:30.743917 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret podName:46289f03-406c-4831-9ad1-0e6751fd7bd5 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:31.243898073 +0000 UTC m=+3.104117061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret") pod "global-pull-secret-syncer-mqv9x" (UID: "46289f03-406c-4831-9ad1-0e6751fd7bd5") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:30.749050 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.744016 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-sysctl-d\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.749050 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.744061 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-var-lib-openvswitch\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.749050 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.744123 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d897418e-880a-46a9-8b13-9c2887bfddb7-cnibin\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.749050 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.744171 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-sys\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.749050 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.744174 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-host\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.749050 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.745415 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f15ba445-e38a-46f6-be0f-65f7307e79d6-agent-certs\") pod \"konnectivity-agent-qjvts\" (UID: \"f15ba445-e38a-46f6-be0f-65f7307e79d6\") " pod="kube-system/konnectivity-agent-qjvts" Oct 30 11:53:30.749050 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.746364 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/554f0b9b-bae8-478b-b3b1-85b376a79cc9-ovn-node-metrics-cert\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.757884 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.757844 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jhg2\" (UniqueName: \"kubernetes.io/projected/617427c7-590d-4d80-8c4e-e11c7812dca0-kube-api-access-5jhg2\") pod \"node-resolver-nhbdl\" (UID: \"617427c7-590d-4d80-8c4e-e11c7812dca0\") " pod="openshift-dns/node-resolver-nhbdl" Oct 30 11:53:30.762438 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.762389 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l95t8\" (UniqueName: \"kubernetes.io/projected/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e-kube-api-access-l95t8\") pod \"node-ca-7c297\" (UID: \"29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e\") " pod="openshift-image-registry/node-ca-7c297" Oct 30 11:53:30.763078 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:30.763032 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:53:30.763183 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:30.763062 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:53:30.763183 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:30.763110 2574 projected.go:196] Error preparing data for projected volume kube-api-access-2jzbb for pod openshift-network-diagnostics/network-check-target-ld8pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:30.763270 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:30.763250 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb podName:363a0adc-193f-48d3-9628-1b9d10665597 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:31.263230504 +0000 UTC m=+3.123449495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2jzbb" (UniqueName: "kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb") pod "network-check-target-ld8pr" (UID: "363a0adc-193f-48d3-9628-1b9d10665597") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:30.764726 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.764678 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8ccv\" (UniqueName: \"kubernetes.io/projected/260d2659-42a8-461f-8994-0c5f9522a80d-kube-api-access-h8ccv\") pod \"network-metrics-daemon-rh7rw\" (UID: \"260d2659-42a8-461f-8994-0c5f9522a80d\") " pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:30.766547 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.766522 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx8v8\" (UniqueName: \"kubernetes.io/projected/b2937779-b03d-4a5b-87bc-f062283eeb2e-kube-api-access-bx8v8\") pod \"iptables-alerter-htqhw\" (UID: \"b2937779-b03d-4a5b-87bc-f062283eeb2e\") " pod="openshift-network-operator/iptables-alerter-htqhw" Oct 30 11:53:30.766651 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.766628 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8tg9\" (UniqueName: \"kubernetes.io/projected/31df51a8-4f22-4745-b25f-8b69654f2845-kube-api-access-v8tg9\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.766992 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.766975 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp7n7\" (UniqueName: \"kubernetes.io/projected/82a77175-bb28-474f-b4ec-dac73d790410-kube-api-access-sp7n7\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.767062 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.766996 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jtv5\" (UniqueName: \"kubernetes.io/projected/554f0b9b-bae8-478b-b3b1-85b376a79cc9-kube-api-access-7jtv5\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.770924 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.770906 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj598\" (UniqueName: \"kubernetes.io/projected/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-kube-api-access-hj598\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.774033 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.774013 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5g55\" (UniqueName: \"kubernetes.io/projected/d897418e-880a-46a9-8b13-9c2887bfddb7-kube-api-access-s5g55\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.835874 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.835840 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7c297" Oct 30 11:53:30.847172 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:30.847143 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d0c5b3_0ef9_4e82_bd58_3fe952dd0d9e.slice/crio-a5a46abc1f84e0d14e43fe38f08c99ad08b17614d7cf8be714cd793b499ded31 WatchSource:0}: Error finding container a5a46abc1f84e0d14e43fe38f08c99ad08b17614d7cf8be714cd793b499ded31: Status 404 returned error can't find the container with id a5a46abc1f84e0d14e43fe38f08c99ad08b17614d7cf8be714cd793b499ded31 Oct 30 11:53:30.858917 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.858895 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:53:30.862916 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.862895 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6pxtt" Oct 30 11:53:30.870171 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:30.870138 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd897418e_880a_46a9_8b13_9c2887bfddb7.slice/crio-cd1ee92b2bc84070ab1e222d12ad2419f4a480741ac772a445913c7eb1e2543c WatchSource:0}: Error finding container cd1ee92b2bc84070ab1e222d12ad2419f4a480741ac772a445913c7eb1e2543c: Status 404 returned error can't find the container with id cd1ee92b2bc84070ab1e222d12ad2419f4a480741ac772a445913c7eb1e2543c Oct 30 11:53:30.879528 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.879504 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nhbdl" Oct 30 11:53:30.885927 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.885908 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:30.889878 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:30.889852 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617427c7_590d_4d80_8c4e_e11c7812dca0.slice/crio-c5c242fec739eaa6bd477b4dbd05b2e5f8b9a8541dd712b1fae043d13dea8674 WatchSource:0}: Error finding container c5c242fec739eaa6bd477b4dbd05b2e5f8b9a8541dd712b1fae043d13dea8674: Status 404 returned error can't find the container with id c5c242fec739eaa6bd477b4dbd05b2e5f8b9a8541dd712b1fae043d13dea8674 Oct 30 11:53:30.892281 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.892259 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:53:30.894466 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.894449 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-htqhw" Oct 30 11:53:30.896823 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:30.896793 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod554f0b9b_bae8_478b_b3b1_85b376a79cc9.slice/crio-54b4e7b90dcf8224029f72fe0f07d265dbfcf68e7392a34072a420cc3cb09f76 WatchSource:0}: Error finding container 54b4e7b90dcf8224029f72fe0f07d265dbfcf68e7392a34072a420cc3cb09f76: Status 404 returned error can't find the container with id 54b4e7b90dcf8224029f72fe0f07d265dbfcf68e7392a34072a420cc3cb09f76 Oct 30 11:53:30.898629 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.898606 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qjvts" Oct 30 11:53:30.905120 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:30.904953 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:53:30.910611 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:30.910582 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2937779_b03d_4a5b_87bc_f062283eeb2e.slice/crio-ba041f17dddd5f9cd8954733ec48b40c3b715d59f841df308916d20f722ca28d WatchSource:0}: Error finding container ba041f17dddd5f9cd8954733ec48b40c3b715d59f841df308916d20f722ca28d: Status 404 returned error can't find the container with id ba041f17dddd5f9cd8954733ec48b40c3b715d59f841df308916d20f722ca28d Oct 30 11:53:30.912408 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:30.912375 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15ba445_e38a_46f6_be0f_65f7307e79d6.slice/crio-e433332d61097a077363fdf206e7b57cbabb4a1b00b56772df19d032b73be9e3 WatchSource:0}: Error finding container e433332d61097a077363fdf206e7b57cbabb4a1b00b56772df19d032b73be9e3: Status 404 returned error can't find the container with id e433332d61097a077363fdf206e7b57cbabb4a1b00b56772df19d032b73be9e3 Oct 30 11:53:30.917185 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:53:30.917165 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f0fad5_3b17_44fe_a2e0_4e99f496ab6e.slice/crio-3e6ade91270c426a9b3776095e9c2fe12a2ad0df284d28264e3491aed57e4c19 WatchSource:0}: Error finding container 3e6ade91270c426a9b3776095e9c2fe12a2ad0df284d28264e3491aed57e4c19: Status 404 returned error can't find the container with id 3e6ade91270c426a9b3776095e9c2fe12a2ad0df284d28264e3491aed57e4c19 Oct 30 11:53:31.336807 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:31.335835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs\") pod \"network-metrics-daemon-rh7rw\" (UID: \"260d2659-42a8-461f-8994-0c5f9522a80d\") " pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:31.336807 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:31.335889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jzbb\" (UniqueName: \"kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb\") pod \"network-check-target-ld8pr\" (UID: \"363a0adc-193f-48d3-9628-1b9d10665597\") " pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:31.336807 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:31.335911 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:31.336807 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:31.336029 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:31.336807 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:31.336098 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret podName:46289f03-406c-4831-9ad1-0e6751fd7bd5 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:32.336078992 +0000 UTC m=+4.196297993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret") pod "global-pull-secret-syncer-mqv9x" (UID: "46289f03-406c-4831-9ad1-0e6751fd7bd5") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:31.336807 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:31.336557 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:31.336807 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:31.336615 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs podName:260d2659-42a8-461f-8994-0c5f9522a80d nodeName:}" failed. No retries permitted until 2026-10-30 11:53:32.336598241 +0000 UTC m=+4.196817231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs") pod "network-metrics-daemon-rh7rw" (UID: "260d2659-42a8-461f-8994-0c5f9522a80d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:31.336807 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:31.336720 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:53:31.336807 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:31.336732 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:53:31.336807 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:31.336741 2574 projected.go:196] Error preparing data for projected volume kube-api-access-2jzbb for pod openshift-network-diagnostics/network-check-target-ld8pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:31.336807 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:31.336772 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb podName:363a0adc-193f-48d3-9628-1b9d10665597 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:32.336760918 +0000 UTC m=+4.196979927 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jzbb" (UniqueName: "kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb") pod "network-check-target-ld8pr" (UID: "363a0adc-193f-48d3-9628-1b9d10665597") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:31.579164 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:31.578877 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:31.580013 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:31.579298 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqv9x" podUID="46289f03-406c-4831-9ad1-0e6751fd7bd5" Oct 30 11:53:31.582154 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:31.582069 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 11:48:31 +0000 UTC" deadline="2028-06-15 10:56:17.091716122 +0000 UTC" Oct 30 11:53:31.582154 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:31.582142 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14255h2m45.509578864s" Oct 30 11:53:31.602404 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:31.602317 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7c297" event={"ID":"29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e","Type":"ContainerStarted","Data":"a5a46abc1f84e0d14e43fe38f08c99ad08b17614d7cf8be714cd793b499ded31"} Oct 30 11:53:31.617172 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:31.616893 2574 generic.go:356] "Generic (PLEG): container finished" podID="ac067dd06bb744f9ffd35d4622ecd99a" containerID="14e8486f873c3bf5d098b8a0d13f2e16042e85a72e0af040ce7ba8dbe87636c7" exitCode=0 Oct 30 11:53:31.617464 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:31.617390 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" event={"ID":"ac067dd06bb744f9ffd35d4622ecd99a","Type":"ContainerDied","Data":"14e8486f873c3bf5d098b8a0d13f2e16042e85a72e0af040ce7ba8dbe87636c7"} Oct 30 11:53:31.632895 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:31.632810 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" event={"ID":"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e","Type":"ContainerStarted","Data":"3e6ade91270c426a9b3776095e9c2fe12a2ad0df284d28264e3491aed57e4c19"} Oct 30 11:53:31.667102 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:31.666992 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-htqhw" event={"ID":"b2937779-b03d-4a5b-87bc-f062283eeb2e","Type":"ContainerStarted","Data":"ba041f17dddd5f9cd8954733ec48b40c3b715d59f841df308916d20f722ca28d"} Oct 30 11:53:31.680011 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:31.679884 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" event={"ID":"554f0b9b-bae8-478b-b3b1-85b376a79cc9","Type":"ContainerStarted","Data":"54b4e7b90dcf8224029f72fe0f07d265dbfcf68e7392a34072a420cc3cb09f76"} Oct 30 11:53:31.684759 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:31.684690 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6pxtt" event={"ID":"31df51a8-4f22-4745-b25f-8b69654f2845","Type":"ContainerStarted","Data":"0b6e449a33b18a7662a942a7cf450cb9b06ddb9743afb5dc310f3422a7cd2dab"} Oct 30 11:53:31.688409 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:31.687792 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal" event={"ID":"5a981c242c496637c2b85dbce301f463","Type":"ContainerStarted","Data":"45de1b8a7e7af25821a185f467e0166bb199a69f0be0feca8643b3166e804e90"} Oct 30 11:53:31.699417 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:31.699382 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qjvts" event={"ID":"f15ba445-e38a-46f6-be0f-65f7307e79d6","Type":"ContainerStarted","Data":"e433332d61097a077363fdf206e7b57cbabb4a1b00b56772df19d032b73be9e3"} Oct 30 11:53:31.709239 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:31.709204 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" event={"ID":"82a77175-bb28-474f-b4ec-dac73d790410","Type":"ContainerStarted","Data":"ecc4d6e3358dd29e7f11102204c36b8bb652a103d91d031c431a2763ced53848"} Oct 30 11:53:31.724743 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:31.724676 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nhbdl" event={"ID":"617427c7-590d-4d80-8c4e-e11c7812dca0","Type":"ContainerStarted","Data":"c5c242fec739eaa6bd477b4dbd05b2e5f8b9a8541dd712b1fae043d13dea8674"} Oct 30 11:53:31.728372 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:31.728297 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7bvp" event={"ID":"d897418e-880a-46a9-8b13-9c2887bfddb7","Type":"ContainerStarted","Data":"cd1ee92b2bc84070ab1e222d12ad2419f4a480741ac772a445913c7eb1e2543c"} Oct 30 11:53:32.346057 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:32.346001 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs\") pod \"network-metrics-daemon-rh7rw\" (UID: \"260d2659-42a8-461f-8994-0c5f9522a80d\") " pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:32.346057 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:32.346056 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jzbb\" (UniqueName: \"kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb\") pod \"network-check-target-ld8pr\" (UID: \"363a0adc-193f-48d3-9628-1b9d10665597\") " pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:32.346336 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:32.346089 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:32.346336 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:32.346236 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:32.346336 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:32.346302 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret podName:46289f03-406c-4831-9ad1-0e6751fd7bd5 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:34.346283132 +0000 UTC m=+6.206502114 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret") pod "global-pull-secret-syncer-mqv9x" (UID: "46289f03-406c-4831-9ad1-0e6751fd7bd5") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:32.346743 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:32.346720 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:32.346863 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:32.346783 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs podName:260d2659-42a8-461f-8994-0c5f9522a80d nodeName:}" failed. No retries permitted until 2026-10-30 11:53:34.346765594 +0000 UTC m=+6.206984568 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs") pod "network-metrics-daemon-rh7rw" (UID: "260d2659-42a8-461f-8994-0c5f9522a80d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:32.346931 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:32.346866 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:53:32.346931 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:32.346885 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:53:32.346931 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:32.346898 2574 projected.go:196] Error preparing data for projected volume kube-api-access-2jzbb for pod openshift-network-diagnostics/network-check-target-ld8pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:32.347078 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:32.346938 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb podName:363a0adc-193f-48d3-9628-1b9d10665597 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:34.346925338 +0000 UTC m=+6.207144315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jzbb" (UniqueName: "kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb") pod "network-check-target-ld8pr" (UID: "363a0adc-193f-48d3-9628-1b9d10665597") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:32.581999 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:32.581264 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:32.581999 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:32.581402 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ld8pr" podUID="363a0adc-193f-48d3-9628-1b9d10665597" Oct 30 11:53:32.581999 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:32.581826 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:32.581999 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:32.581937 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh7rw" podUID="260d2659-42a8-461f-8994-0c5f9522a80d" Oct 30 11:53:32.582844 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:32.582768 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 11:48:31 +0000 UTC" deadline="2028-04-11 12:27:13.269592505 +0000 UTC" Oct 30 11:53:32.582844 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:32.582798 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12696h33m40.686798425s" Oct 30 11:53:32.743670 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:32.743623 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" event={"ID":"ac067dd06bb744f9ffd35d4622ecd99a","Type":"ContainerStarted","Data":"f911c175e62302e81ef9da5ce4089af7daf6745fc776f535f6b8902071ca0e01"} Oct 30 11:53:32.791996 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:32.791928 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal" podStartSLOduration=2.791904444 podStartE2EDuration="2.791904444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:31.732987395 +0000 UTC m=+3.593206395" watchObservedRunningTime="2026-10-30 11:53:32.791904444 +0000 UTC m=+4.652123446" Oct 30 11:53:33.578682 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:33.578137 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:33.578682 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:33.578300 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqv9x" podUID="46289f03-406c-4831-9ad1-0e6751fd7bd5" Oct 30 11:53:33.753204 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:33.751775 2574 generic.go:356] "Generic (PLEG): container finished" podID="d897418e-880a-46a9-8b13-9c2887bfddb7" containerID="5e6e6fd4e22259a168f79fa076929dda8721e979ddfebb85d211563113410389" exitCode=0 Oct 30 11:53:33.753204 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:33.752904 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7bvp" event={"ID":"d897418e-880a-46a9-8b13-9c2887bfddb7","Type":"ContainerDied","Data":"5e6e6fd4e22259a168f79fa076929dda8721e979ddfebb85d211563113410389"} Oct 30 11:53:33.805311 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:33.805231 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" podStartSLOduration=3.805210023 podStartE2EDuration="3.805210023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:32.792653787 +0000 UTC m=+4.652872791" watchObservedRunningTime="2026-10-30 11:53:33.805210023 +0000 UTC m=+5.665429023" Oct 30 11:53:34.364774 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:34.364706 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs\") pod \"network-metrics-daemon-rh7rw\" (UID: \"260d2659-42a8-461f-8994-0c5f9522a80d\") " pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:34.364977 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:34.364792 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jzbb\" (UniqueName: \"kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb\") pod \"network-check-target-ld8pr\" (UID: \"363a0adc-193f-48d3-9628-1b9d10665597\") " pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:34.364977 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:34.364826 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:34.364977 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:34.364971 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:34.365170 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:34.365039 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret podName:46289f03-406c-4831-9ad1-0e6751fd7bd5 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:38.365020575 +0000 UTC m=+10.225239555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret") pod "global-pull-secret-syncer-mqv9x" (UID: "46289f03-406c-4831-9ad1-0e6751fd7bd5") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:34.365518 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:34.365485 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:34.365616 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:34.365544 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs podName:260d2659-42a8-461f-8994-0c5f9522a80d nodeName:}" failed. No retries permitted until 2026-10-30 11:53:38.365527522 +0000 UTC m=+10.225746500 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs") pod "network-metrics-daemon-rh7rw" (UID: "260d2659-42a8-461f-8994-0c5f9522a80d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:34.365684 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:34.365624 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:53:34.365684 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:34.365638 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:53:34.365684 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:34.365651 2574 projected.go:196] Error preparing data for projected volume kube-api-access-2jzbb for pod openshift-network-diagnostics/network-check-target-ld8pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:34.365891 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:34.365687 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb podName:363a0adc-193f-48d3-9628-1b9d10665597 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:38.365675717 +0000 UTC m=+10.225894696 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jzbb" (UniqueName: "kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb") pod "network-check-target-ld8pr" (UID: "363a0adc-193f-48d3-9628-1b9d10665597") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:34.585796 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:34.584465 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:34.585796 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:34.584749 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh7rw" podUID="260d2659-42a8-461f-8994-0c5f9522a80d" Oct 30 11:53:34.585796 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:34.584944 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:34.585796 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:34.585446 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ld8pr" podUID="363a0adc-193f-48d3-9628-1b9d10665597" Oct 30 11:53:35.578323 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:35.578285 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:35.578826 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:35.578438 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqv9x" podUID="46289f03-406c-4831-9ad1-0e6751fd7bd5" Oct 30 11:53:35.764820 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:35.764779 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qjvts" event={"ID":"f15ba445-e38a-46f6-be0f-65f7307e79d6","Type":"ContainerStarted","Data":"abbc433e7e22252d8b3ad7c66fb445878606a5013b0a688319f48551c7e83808"} Oct 30 11:53:36.581348 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:36.580298 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:36.581348 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:36.580454 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh7rw" podUID="260d2659-42a8-461f-8994-0c5f9522a80d" Oct 30 11:53:36.581348 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:36.580894 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:36.581348 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:36.580989 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ld8pr" podUID="363a0adc-193f-48d3-9628-1b9d10665597" Oct 30 11:53:36.774322 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:36.774273 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7c297" event={"ID":"29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e","Type":"ContainerStarted","Data":"4dae001fbdec39c7235e5db71e6caf5dfebad928b7acfff06e9c163286bf25a7"} Oct 30 11:53:36.820278 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:36.819305 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qjvts" podStartSLOduration=3.646603226 podStartE2EDuration="7.819284919s" totalImagesPullingTime="4.172681693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:35.811598049 +0000 UTC m=+7.671817048" watchObservedRunningTime="2026-10-30 11:53:36.819284919 +0000 UTC m=+8.679503919" Oct 30 11:53:36.820278 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:36.819497 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7c297" podStartSLOduration=2.808207617 podStartE2EDuration="7.819483852s" totalImagesPullingTime="5.011276235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:36.818993329 +0000 UTC m=+8.679212342" watchObservedRunningTime="2026-10-30 11:53:36.819483852 +0000 UTC m=+8.679702853" Oct 30 11:53:36.950890 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:36.950854 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qjvts" Oct 30 11:53:36.951430 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:36.951410 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qjvts" Oct 30 11:53:37.578631 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:37.578524 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:37.578816 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:37.578685 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqv9x" podUID="46289f03-406c-4831-9ad1-0e6751fd7bd5" Oct 30 11:53:37.781967 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:37.781411 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" event={"ID":"82a77175-bb28-474f-b4ec-dac73d790410","Type":"ContainerStarted","Data":"12bad173ac1d665a9193ea84cfd5f6e394a0cdacc17bc9553bb0a3157d87833f"} Oct 30 11:53:37.782499 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:37.782019 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qjvts" Oct 30 11:53:37.782640 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:37.782595 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qjvts" Oct 30 11:53:38.403472 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:38.403269 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs\") pod \"network-metrics-daemon-rh7rw\" (UID: \"260d2659-42a8-461f-8994-0c5f9522a80d\") " pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:38.404593 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:38.403766 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jzbb\" (UniqueName: \"kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb\") pod \"network-check-target-ld8pr\" (UID: \"363a0adc-193f-48d3-9628-1b9d10665597\") " pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:38.404593 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:38.403812 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:38.404593 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:38.403449 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:38.404593 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:38.403997 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs podName:260d2659-42a8-461f-8994-0c5f9522a80d nodeName:}" failed. No retries permitted until 2026-10-30 11:53:46.403979496 +0000 UTC m=+18.264198476 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs") pod "network-metrics-daemon-rh7rw" (UID: "260d2659-42a8-461f-8994-0c5f9522a80d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:38.404593 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:38.404441 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:53:38.404593 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:38.404466 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:53:38.404593 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:38.404479 2574 projected.go:196] Error preparing data for projected volume kube-api-access-2jzbb for pod openshift-network-diagnostics/network-check-target-ld8pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:38.404593 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:38.404524 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb podName:363a0adc-193f-48d3-9628-1b9d10665597 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:46.404509219 +0000 UTC m=+18.264728197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jzbb" (UniqueName: "kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb") pod "network-check-target-ld8pr" (UID: "363a0adc-193f-48d3-9628-1b9d10665597") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:38.404593 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:38.403930 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:38.404593 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:38.404565 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret podName:46289f03-406c-4831-9ad1-0e6751fd7bd5 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:46.404553727 +0000 UTC m=+18.264772704 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret") pod "global-pull-secret-syncer-mqv9x" (UID: "46289f03-406c-4831-9ad1-0e6751fd7bd5") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:38.580741 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:38.579274 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:38.580741 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:38.579429 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh7rw" podUID="260d2659-42a8-461f-8994-0c5f9522a80d" Oct 30 11:53:38.581613 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:38.580881 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:38.581613 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:38.580986 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ld8pr" podUID="363a0adc-193f-48d3-9628-1b9d10665597" Oct 30 11:53:38.673453 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:38.673418 2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 11:53:38.788737 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:38.788690 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" event={"ID":"82a77175-bb28-474f-b4ec-dac73d790410","Type":"ContainerStarted","Data":"86cb12c0c69320ea71349abf0177f907c0a382d93b6b3a4f120e6f4277d2dffe"} Oct 30 11:53:38.796981 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:38.796941 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7bvp" event={"ID":"d897418e-880a-46a9-8b13-9c2887bfddb7","Type":"ContainerStarted","Data":"36d3aec6dee2041d7691fa089efced35c8034a425e6ab7cd14af6d6afce36fa3"} Oct 30 11:53:39.578571 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:39.578484 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:39.578754 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:39.578629 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqv9x" podUID="46289f03-406c-4831-9ad1-0e6751fd7bd5" Oct 30 11:53:39.595560 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:39.595449 2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T11:53:38.673449585Z","UUID":"3e7086a0-5c10-4dd9-898d-a9c9fd620597","Handler":null,"Name":"","Endpoint":""} Oct 30 11:53:39.597324 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:39.597299 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 11:53:39.597446 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:39.597334 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 11:53:39.800588 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:39.800547 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nhbdl" event={"ID":"617427c7-590d-4d80-8c4e-e11c7812dca0","Type":"ContainerStarted","Data":"70cc95f7711e13f057efdd94b47df0623fd769d09d8f1705d143fd4bdf55f2a3"} Oct 30 11:53:39.804292 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:39.803552 2574 generic.go:356] "Generic (PLEG): container finished" podID="d897418e-880a-46a9-8b13-9c2887bfddb7" containerID="36d3aec6dee2041d7691fa089efced35c8034a425e6ab7cd14af6d6afce36fa3" exitCode=0 Oct 30 11:53:39.804292 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:39.803656 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7bvp" event={"ID":"d897418e-880a-46a9-8b13-9c2887bfddb7","Type":"ContainerDied","Data":"36d3aec6dee2041d7691fa089efced35c8034a425e6ab7cd14af6d6afce36fa3"} Oct 30 11:53:39.807336 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:39.807232 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6pxtt" event={"ID":"31df51a8-4f22-4745-b25f-8b69654f2845","Type":"ContainerStarted","Data":"e9dfb5ad92e6793b14f5cde667549c297775c737e6f9fd4902e04d136e4e4329"} Oct 30 11:53:39.846900 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:39.846848 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-nhbdl" podStartSLOduration=2.579816235 podStartE2EDuration="10.846833805s" totalImagesPullingTime="8.26701757s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:39.846444925 +0000 UTC m=+11.706663925" watchObservedRunningTime="2026-10-30 11:53:39.846833805 +0000 UTC m=+11.707052805" Oct 30 11:53:40.578941 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:40.578913 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:40.579090 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:40.579050 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ld8pr" podUID="363a0adc-193f-48d3-9628-1b9d10665597" Oct 30 11:53:40.579175 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:40.579157 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:40.579267 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:40.579248 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh7rw" podUID="260d2659-42a8-461f-8994-0c5f9522a80d" Oct 30 11:53:40.815413 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:40.815369 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" event={"ID":"82a77175-bb28-474f-b4ec-dac73d790410","Type":"ContainerStarted","Data":"d09304c2a8f1c6bb819a6886c967718908500df44ae3bff55dd8b14c230744f8"} Oct 30 11:53:40.818671 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:40.818424 2574 generic.go:356] "Generic (PLEG): container finished" podID="d897418e-880a-46a9-8b13-9c2887bfddb7" containerID="06375939516acac61587d3a01ee9a13dbf948dae058996d53432921f44a71f34" exitCode=0 Oct 30 11:53:40.818671 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:40.818546 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7bvp" event={"ID":"d897418e-880a-46a9-8b13-9c2887bfddb7","Type":"ContainerDied","Data":"06375939516acac61587d3a01ee9a13dbf948dae058996d53432921f44a71f34"} Oct 30 11:53:40.865505 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:40.865435 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-6pxtt" podStartSLOduration=3.857073699 podStartE2EDuration="11.865412555s" totalImagesPullingTime="8.008338856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:39.951521073 +0000 UTC m=+11.811740074" watchObservedRunningTime="2026-10-30 11:53:40.865412555 +0000 UTC m=+12.725631554" Oct 30 11:53:40.920552 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:40.920488 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" podStartSLOduration=3.426169919 podStartE2EDuration="11.920466144s" totalImagesPullingTime="8.494296225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:40.865799908 +0000 UTC m=+12.726018919" watchObservedRunningTime="2026-10-30 11:53:40.920466144 +0000 UTC m=+12.780685143" Oct 30 11:53:41.577947 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:41.577925 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:41.578086 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:41.578043 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqv9x" podUID="46289f03-406c-4831-9ad1-0e6751fd7bd5" Oct 30 11:53:41.821575 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:41.821500 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" event={"ID":"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e","Type":"ContainerStarted","Data":"80a4b38eb325c5eee62f3a20adc28e5fa92e0315938930f1291bf47ce7b352c9"} Oct 30 11:53:41.822981 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:41.822957 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-htqhw" event={"ID":"b2937779-b03d-4a5b-87bc-f062283eeb2e","Type":"ContainerStarted","Data":"8c59e6316ee7409271bda6f8c902aec89af9c7b7e75d414c604a7690c949f7df"} Oct 30 11:53:41.866637 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:41.866579 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" podStartSLOduration=2.248542946 podStartE2EDuration="12.866561881s" totalImagesPullingTime="10.618018935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:41.86645272 +0000 UTC m=+13.726671717" watchObservedRunningTime="2026-10-30 11:53:41.866561881 +0000 UTC m=+13.726780879" Oct 30 11:53:41.911171 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:41.911117 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-htqhw" podStartSLOduration=4.517769641 podStartE2EDuration="12.91109835s" totalImagesPullingTime="8.393328709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:41.910152021 +0000 UTC m=+13.770371018" watchObservedRunningTime="2026-10-30 11:53:41.91109835 +0000 UTC m=+13.771317346" Oct 30 11:53:42.578551 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:42.578510 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:42.578551 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:42.578558 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:42.578774 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:42.578681 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh7rw" podUID="260d2659-42a8-461f-8994-0c5f9522a80d" Oct 30 11:53:42.578831 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:42.578785 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ld8pr" podUID="363a0adc-193f-48d3-9628-1b9d10665597" Oct 30 11:53:42.827977 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:42.827934 2574 generic.go:356] "Generic (PLEG): container finished" podID="d897418e-880a-46a9-8b13-9c2887bfddb7" containerID="2037fa397d756b817c26e761a4f174416df9e18f905d2f93316da1cb9a595cf7" exitCode=0 Oct 30 11:53:42.828594 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:42.828103 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7bvp" event={"ID":"d897418e-880a-46a9-8b13-9c2887bfddb7","Type":"ContainerDied","Data":"2037fa397d756b817c26e761a4f174416df9e18f905d2f93316da1cb9a595cf7"} Oct 30 11:53:43.577917 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:43.577881 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:43.578125 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:43.578036 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqv9x" podUID="46289f03-406c-4831-9ad1-0e6751fd7bd5" Oct 30 11:53:44.578566 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:44.578368 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:44.578971 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:44.578439 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:44.578971 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:44.578687 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh7rw" podUID="260d2659-42a8-461f-8994-0c5f9522a80d" Oct 30 11:53:44.578971 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:44.578743 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ld8pr" podUID="363a0adc-193f-48d3-9628-1b9d10665597" Oct 30 11:53:44.837143 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:44.837009 2574 generic.go:356] "Generic (PLEG): container finished" podID="d897418e-880a-46a9-8b13-9c2887bfddb7" containerID="40ca657fc904a45b0a291858c0054f6b5194b92c21a0309f77e9f6e7233b454a" exitCode=0 Oct 30 11:53:44.837143 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:44.837096 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7bvp" event={"ID":"d897418e-880a-46a9-8b13-9c2887bfddb7","Type":"ContainerDied","Data":"40ca657fc904a45b0a291858c0054f6b5194b92c21a0309f77e9f6e7233b454a"} Oct 30 11:53:45.578229 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:45.578192 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:45.578399 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:45.578299 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqv9x" podUID="46289f03-406c-4831-9ad1-0e6751fd7bd5" Oct 30 11:53:45.841699 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:45.841608 2574 generic.go:356] "Generic (PLEG): container finished" podID="d897418e-880a-46a9-8b13-9c2887bfddb7" containerID="e3d17439b9adf04c3d12a7140cf0a85ef10f0718789d076e25c554df31ded05e" exitCode=0 Oct 30 11:53:45.841699 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:45.841674 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7bvp" event={"ID":"d897418e-880a-46a9-8b13-9c2887bfddb7","Type":"ContainerDied","Data":"e3d17439b9adf04c3d12a7140cf0a85ef10f0718789d076e25c554df31ded05e"} Oct 30 11:53:46.463883 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:46.463833 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs\") pod \"network-metrics-daemon-rh7rw\" (UID: \"260d2659-42a8-461f-8994-0c5f9522a80d\") " pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:46.463883 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:46.463879 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jzbb\" (UniqueName: \"kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb\") pod \"network-check-target-ld8pr\" (UID: \"363a0adc-193f-48d3-9628-1b9d10665597\") " pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:46.464150 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:46.463907 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:46.464150 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:46.464003 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:46.464150 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:46.464035 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:53:46.464150 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:46.464056 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:53:46.464150 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:46.464064 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:46.464150 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:46.464096 2574 projected.go:196] Error preparing data for projected volume kube-api-access-2jzbb for pod openshift-network-diagnostics/network-check-target-ld8pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:46.464150 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:46.464101 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs podName:260d2659-42a8-461f-8994-0c5f9522a80d nodeName:}" failed. No retries permitted until 2026-10-30 11:54:02.464079805 +0000 UTC m=+34.324298796 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs") pod "network-metrics-daemon-rh7rw" (UID: "260d2659-42a8-461f-8994-0c5f9522a80d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:46.464150 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:46.464128 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret podName:46289f03-406c-4831-9ad1-0e6751fd7bd5 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:02.46411279 +0000 UTC m=+34.324331776 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret") pod "global-pull-secret-syncer-mqv9x" (UID: "46289f03-406c-4831-9ad1-0e6751fd7bd5") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:46.464150 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:46.464158 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb podName:363a0adc-193f-48d3-9628-1b9d10665597 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:02.464148778 +0000 UTC m=+34.324367751 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jzbb" (UniqueName: "kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb") pod "network-check-target-ld8pr" (UID: "363a0adc-193f-48d3-9628-1b9d10665597") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:46.578649 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:46.578622 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:46.578818 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:46.578661 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:46.578818 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:46.578762 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh7rw" podUID="260d2659-42a8-461f-8994-0c5f9522a80d" Oct 30 11:53:46.578908 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:46.578843 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ld8pr" podUID="363a0adc-193f-48d3-9628-1b9d10665597" Oct 30 11:53:46.847152 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:46.847108 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7bvp" event={"ID":"d897418e-880a-46a9-8b13-9c2887bfddb7","Type":"ContainerStarted","Data":"e00de4c29d0e789cd3c00a88f2082e60a8de00ebc39f59774e8545cd0af56b01"} Oct 30 11:53:46.849745 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:46.849722 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" event={"ID":"554f0b9b-bae8-478b-b3b1-85b376a79cc9","Type":"ContainerStarted","Data":"509f2de18ad33fed270707ea19697e8a210b638e7e02d5dd65cb9d96c2fcee7c"} Oct 30 11:53:46.849861 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:46.849784 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" event={"ID":"554f0b9b-bae8-478b-b3b1-85b376a79cc9","Type":"ContainerStarted","Data":"e8dec5f9ce06ed57036ef04684ceb080f20847bbb04251740d7ff16de444df7c"} Oct 30 11:53:46.849861 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:46.849816 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" event={"ID":"554f0b9b-bae8-478b-b3b1-85b376a79cc9","Type":"ContainerStarted","Data":"dbba6a98a9975b930e4a78ae5d1e4f20702e8573091b358bcd891719d7f5debd"} Oct 30 11:53:46.849861 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:46.849842 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" event={"ID":"554f0b9b-bae8-478b-b3b1-85b376a79cc9","Type":"ContainerStarted","Data":"5ee05ec6d360294350b55c966878fee0278c71d85a36084935390a617314c959"} Oct 30 11:53:46.849956 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:46.849878 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" event={"ID":"554f0b9b-bae8-478b-b3b1-85b376a79cc9","Type":"ContainerStarted","Data":"b3d48f6943a1156a08fa05706792473f43aff664329a6b379f2b84950de2fbdb"} Oct 30 11:53:46.849956 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:46.849908 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" event={"ID":"554f0b9b-bae8-478b-b3b1-85b376a79cc9","Type":"ContainerStarted","Data":"6267a0b35953d6a8ce2a460d4398d3d1c9c88d9a54ebaf6f754e5c21c4cd5f5e"} Oct 30 11:53:46.896759 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:46.896712 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d7bvp" podStartSLOduration=7.465490476 podStartE2EDuration="17.896698842s" totalImagesPullingTime="10.431208366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:46.896461017 +0000 UTC m=+18.756680013" watchObservedRunningTime="2026-10-30 11:53:46.896698842 +0000 UTC m=+18.756917836" Oct 30 11:53:47.577767 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:47.577715 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:47.577928 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:47.577838 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqv9x" podUID="46289f03-406c-4831-9ad1-0e6751fd7bd5" Oct 30 11:53:48.581245 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:48.581213 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:48.581869 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:48.581213 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:48.581869 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:48.581314 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh7rw" podUID="260d2659-42a8-461f-8994-0c5f9522a80d" Oct 30 11:53:48.581869 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:48.581409 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ld8pr" podUID="363a0adc-193f-48d3-9628-1b9d10665597" Oct 30 11:53:48.857765 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:48.857685 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" event={"ID":"554f0b9b-bae8-478b-b3b1-85b376a79cc9","Type":"ContainerStarted","Data":"b1f4b2e749bde58c7bd26232f9217e9038723a9c6392f0bc1fa5a00a5879d483"} Oct 30 11:53:49.578674 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:49.578641 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:49.578837 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:49.578777 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqv9x" podUID="46289f03-406c-4831-9ad1-0e6751fd7bd5" Oct 30 11:53:50.581980 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:50.581952 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:50.582677 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:50.582110 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh7rw" podUID="260d2659-42a8-461f-8994-0c5f9522a80d" Oct 30 11:53:50.582677 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:50.582149 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:50.582677 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:50.582232 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ld8pr" podUID="363a0adc-193f-48d3-9628-1b9d10665597" Oct 30 11:53:50.865505 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:50.865426 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" event={"ID":"554f0b9b-bae8-478b-b3b1-85b376a79cc9","Type":"ContainerStarted","Data":"ab4373f39fc3f7f0d2015acb8d5e5de44bab5bffd90e3386acd5c813d8c02575"} Oct 30 11:53:50.865801 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:50.865782 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:50.880935 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:50.880908 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:50.960196 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:50.960144 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" podStartSLOduration=6.804341749 podStartE2EDuration="21.960131861s" totalImagesPullingTime="15.155790112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:50.959630265 +0000 UTC m=+22.819849261" watchObservedRunningTime="2026-10-30 11:53:50.960131861 +0000 UTC m=+22.820350856" Oct 30 11:53:51.577856 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:51.577827 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:51.578016 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:51.577931 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqv9x" podUID="46289f03-406c-4831-9ad1-0e6751fd7bd5" Oct 30 11:53:51.868251 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:51.868164 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:51.869977 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:51.869173 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:51.895366 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:51.886556 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:53:52.581287 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:52.581230 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:52.581437 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:52.581359 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ld8pr" podUID="363a0adc-193f-48d3-9628-1b9d10665597" Oct 30 11:53:52.581437 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:52.581396 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:52.581519 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:52.581470 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh7rw" podUID="260d2659-42a8-461f-8994-0c5f9522a80d" Oct 30 11:53:52.638836 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:52.638809 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ld8pr"] Oct 30 11:53:52.643608 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:52.643581 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mqv9x"] Oct 30 11:53:52.643742 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:52.643705 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:52.643843 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:52.643822 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqv9x" podUID="46289f03-406c-4831-9ad1-0e6751fd7bd5" Oct 30 11:53:52.644285 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:52.644268 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rh7rw"] Oct 30 11:53:52.869801 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:52.869731 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:52.870193 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:52.869736 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:52.870193 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:52.869829 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh7rw" podUID="260d2659-42a8-461f-8994-0c5f9522a80d" Oct 30 11:53:52.870193 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:52.869896 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ld8pr" podUID="363a0adc-193f-48d3-9628-1b9d10665597" Oct 30 11:53:54.578870 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:54.578834 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:54.579322 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:54.578834 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:54.579322 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:54.578954 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqv9x" podUID="46289f03-406c-4831-9ad1-0e6751fd7bd5" Oct 30 11:53:54.579322 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:54.578834 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:54.579322 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:54.579056 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh7rw" podUID="260d2659-42a8-461f-8994-0c5f9522a80d" Oct 30 11:53:54.579322 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:54.579180 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ld8pr" podUID="363a0adc-193f-48d3-9628-1b9d10665597" Oct 30 11:53:56.578357 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:56.578320 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:56.578767 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:56.578321 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:56.578767 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:56.578420 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqv9x" podUID="46289f03-406c-4831-9ad1-0e6751fd7bd5" Oct 30 11:53:56.578767 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:56.578447 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:56.578767 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:56.578553 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh7rw" podUID="260d2659-42a8-461f-8994-0c5f9522a80d" Oct 30 11:53:56.578767 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:56.578642 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ld8pr" podUID="363a0adc-193f-48d3-9628-1b9d10665597" Oct 30 11:53:58.579519 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:58.579486 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:53:58.579999 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:58.579580 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqv9x" podUID="46289f03-406c-4831-9ad1-0e6751fd7bd5" Oct 30 11:53:58.579999 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:58.579626 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:53:58.579999 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:58.579667 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ld8pr" podUID="363a0adc-193f-48d3-9628-1b9d10665597" Oct 30 11:53:58.579999 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:58.579694 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:53:58.579999 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:58.579751 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh7rw" podUID="260d2659-42a8-461f-8994-0c5f9522a80d" Oct 30 11:53:59.362953 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.362925 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeReady" Oct 30 11:53:59.363136 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.363040 2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 11:53:59.439087 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.439047 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-z6hh6"] Oct 30 11:53:59.470974 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.470523 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p4nvb"] Oct 30 11:53:59.470974 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.470640 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z6hh6" Oct 30 11:53:59.479536 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.479514 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p4nvb"] Oct 30 11:53:59.479638 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.479548 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z6hh6"] Oct 30 11:53:59.479638 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.479622 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p4nvb" Oct 30 11:53:59.496196 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.496172 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6dvql"] Oct 30 11:53:59.507843 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.507819 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:53:59.523221 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.523172 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6dvql"] Oct 30 11:53:59.538750 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.538726 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 11:53:59.539124 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.539110 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 11:53:59.539338 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.539323 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjl8c\"" Oct 30 11:53:59.539392 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.539327 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tdxpc\"" Oct 30 11:53:59.539392 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.539341 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8pg5x\"" Oct 30 11:53:59.539763 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.539724 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 11:53:59.539763 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.539727 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 11:53:59.539763 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.539752 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 11:53:59.539763 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.539730 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 11:53:59.540018 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.539756 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 11:53:59.540018 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.539790 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 11:53:59.540136 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.540030 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 11:53:59.558187 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.558159 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a948657-c616-435c-90db-b880a0db5bd9-config-volume\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:53:59.558324 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.558195 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmctv\" (UniqueName: \"kubernetes.io/projected/1a948657-c616-435c-90db-b880a0db5bd9-kube-api-access-lmctv\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:53:59.558324 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.558213 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a948657-c616-435c-90db-b880a0db5bd9-tmp-dir\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:53:59.558411 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.558374 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:53:59.558411 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.558403 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d23976a-d44f-4b71-8b58-04932007cc17-cert\") pod \"ingress-canary-z6hh6\" (UID: \"9d23976a-d44f-4b71-8b58-04932007cc17\") " pod="openshift-ingress-canary/ingress-canary-z6hh6" Oct 30 11:53:59.558478 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.558423 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p7nr\" (UniqueName: \"kubernetes.io/projected/9d23976a-d44f-4b71-8b58-04932007cc17-kube-api-access-5p7nr\") pod \"ingress-canary-z6hh6\" (UID: \"9d23976a-d44f-4b71-8b58-04932007cc17\") " pod="openshift-ingress-canary/ingress-canary-z6hh6" Oct 30 11:53:59.588744 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.588718 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-nr74p-m5v7z"] Oct 30 11:53:59.612053 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.612023 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:53:59.612635 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.612614 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-nr74p-m5v7z"] Oct 30 11:53:59.641233 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.641209 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-96ksc\"" Oct 30 11:53:59.641644 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.641628 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 11:53:59.656205 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.656176 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 11:53:59.658865 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.658846 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:53:59.658940 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.658877 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d23976a-d44f-4b71-8b58-04932007cc17-cert\") pod \"ingress-canary-z6hh6\" (UID: \"9d23976a-d44f-4b71-8b58-04932007cc17\") " pod="openshift-ingress-canary/ingress-canary-z6hh6" Oct 30 11:53:59.658940 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.658903 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f2f3c38-f710-491a-9728-7c641291d679-crio-socket\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:53:59.658940 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.658929 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:53:59.659086 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.658955 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxpq6\" (UniqueName: \"kubernetes.io/projected/4f2f3c38-f710-491a-9728-7c641291d679-kube-api-access-nxpq6\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:53:59.659086 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.658999 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p7nr\" (UniqueName: \"kubernetes.io/projected/9d23976a-d44f-4b71-8b58-04932007cc17-kube-api-access-5p7nr\") pod \"ingress-canary-z6hh6\" (UID: \"9d23976a-d44f-4b71-8b58-04932007cc17\") " pod="openshift-ingress-canary/ingress-canary-z6hh6" Oct 30 11:53:59.659086 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.659027 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f2f3c38-f710-491a-9728-7c641291d679-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:53:59.659086 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:59.659032 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:53:59.659273 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:59.659100 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls podName:1a948657-c616-435c-90db-b880a0db5bd9 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:00.159084659 +0000 UTC m=+32.019303637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls") pod "dns-default-p4nvb" (UID: "1a948657-c616-435c-90db-b880a0db5bd9") : secret "dns-default-metrics-tls" not found Oct 30 11:53:59.659273 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.659162 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a948657-c616-435c-90db-b880a0db5bd9-config-volume\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:53:59.659273 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.659198 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f2f3c38-f710-491a-9728-7c641291d679-data-volume\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:53:59.659273 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.659224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmctv\" (UniqueName: \"kubernetes.io/projected/1a948657-c616-435c-90db-b880a0db5bd9-kube-api-access-lmctv\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:53:59.659427 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.659273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a948657-c616-435c-90db-b880a0db5bd9-tmp-dir\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:53:59.659547 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.659532 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a948657-c616-435c-90db-b880a0db5bd9-tmp-dir\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:53:59.659722 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.659706 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a948657-c616-435c-90db-b880a0db5bd9-config-volume\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:53:59.662710 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.662694 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d23976a-d44f-4b71-8b58-04932007cc17-cert\") pod \"ingress-canary-z6hh6\" (UID: \"9d23976a-d44f-4b71-8b58-04932007cc17\") " pod="openshift-ingress-canary/ingress-canary-z6hh6" Oct 30 11:53:59.690891 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.690863 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmctv\" (UniqueName: \"kubernetes.io/projected/1a948657-c616-435c-90db-b880a0db5bd9-kube-api-access-lmctv\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:53:59.691256 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.691238 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p7nr\" (UniqueName: \"kubernetes.io/projected/9d23976a-d44f-4b71-8b58-04932007cc17-kube-api-access-5p7nr\") pod \"ingress-canary-z6hh6\" (UID: \"9d23976a-d44f-4b71-8b58-04932007cc17\") " pod="openshift-ingress-canary/ingress-canary-z6hh6" Oct 30 11:53:59.760406 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.760370 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f2f3c38-f710-491a-9728-7c641291d679-crio-socket\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:53:59.760406 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.760407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:53:59.760634 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.760427 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpq6\" (UniqueName: \"kubernetes.io/projected/4f2f3c38-f710-491a-9728-7c641291d679-kube-api-access-nxpq6\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:53:59.760634 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.760487 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lswd\" (UniqueName: \"kubernetes.io/projected/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-kube-api-access-6lswd\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:53:59.760634 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.760524 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f2f3c38-f710-491a-9728-7c641291d679-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:53:59.760634 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:59.760551 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:53:59.760634 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.760603 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-trusted-ca-bundle\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:53:59.760634 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.760620 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f2f3c38-f710-491a-9728-7c641291d679-crio-socket\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:53:59.760918 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:59.760635 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls podName:4f2f3c38-f710-491a-9728-7c641291d679 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:00.26061542 +0000 UTC m=+32.120834397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dvql" (UID: "4f2f3c38-f710-491a-9728-7c641291d679") : secret "insights-runtime-extractor-tls" not found Oct 30 11:53:59.760918 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:59.760664 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 11:53:59.760918 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:53:59.760766 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4f2f3c38-f710-491a-9728-7c641291d679-kube-rbac-proxy-cm podName:4f2f3c38-f710-491a-9728-7c641291d679 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:00.260752914 +0000 UTC m=+32.120971892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4f2f3c38-f710-491a-9728-7c641291d679-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6dvql" (UID: "4f2f3c38-f710-491a-9728-7c641291d679") : configmap "kube-rbac-proxy" not found Oct 30 11:53:59.760918 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.760782 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f2f3c38-f710-491a-9728-7c641291d679-data-volume\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:53:59.760918 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.760811 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-service-ca-bundle\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:53:59.760918 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.760850 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-archives-path\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:53:59.761126 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.761064 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f2f3c38-f710-491a-9728-7c641291d679-data-volume\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:53:59.788622 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.788566 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z6hh6" Oct 30 11:53:59.793421 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.793397 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxpq6\" (UniqueName: \"kubernetes.io/projected/4f2f3c38-f710-491a-9728-7c641291d679-kube-api-access-nxpq6\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:53:59.861818 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.861783 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lswd\" (UniqueName: \"kubernetes.io/projected/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-kube-api-access-6lswd\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:53:59.861993 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.861828 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-trusted-ca-bundle\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:53:59.861993 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.861854 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-service-ca-bundle\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:53:59.862122 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.862018 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-archives-path\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:53:59.862364 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.862345 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-archives-path\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:53:59.862422 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.862406 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-service-ca-bundle\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:53:59.862758 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.862741 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-trusted-ca-bundle\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:53:59.890469 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.889634 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lswd\" (UniqueName: \"kubernetes.io/projected/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-kube-api-access-6lswd\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:53:59.935220 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:53:59.935186 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:54:00.135955 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.135932 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z6hh6"] Oct 30 11:54:00.137238 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:54:00.137215 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d23976a_d44f_4b71_8b58_04932007cc17.slice/crio-8924ad7e7609d36b9309221ed6f63cf95f9beb316abade30a75232ef6b4f1000 WatchSource:0}: Error finding container 8924ad7e7609d36b9309221ed6f63cf95f9beb316abade30a75232ef6b4f1000: Status 404 returned error can't find the container with id 8924ad7e7609d36b9309221ed6f63cf95f9beb316abade30a75232ef6b4f1000 Oct 30 11:54:00.164460 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.164429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:54:00.164597 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:00.164579 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:54:00.164649 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:00.164640 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls podName:1a948657-c616-435c-90db-b880a0db5bd9 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:01.164625787 +0000 UTC m=+33.024844764 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls") pod "dns-default-p4nvb" (UID: "1a948657-c616-435c-90db-b880a0db5bd9") : secret "dns-default-metrics-tls" not found Oct 30 11:54:00.191303 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.191272 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-gqgxk"] Oct 30 11:54:00.220877 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.220850 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-gqgxk"] Oct 30 11:54:00.221002 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.220967 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" Oct 30 11:54:00.239406 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.239375 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 11:54:00.239793 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.239683 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 11:54:00.239793 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.239718 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 11:54:00.239793 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.239718 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 11:54:00.239793 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.239744 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-847hl\"" Oct 30 11:54:00.240640 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.240622 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-nr74p-m5v7z"] Oct 30 11:54:00.241447 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:54:00.241424 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0f7768_5ba5_4b50_a601_0c4202b2b66e.slice/crio-3545637cdbb817797ebdd7286acb317ed11a1a4a4daf67cc766df1471b9a3c50 WatchSource:0}: Error finding container 3545637cdbb817797ebdd7286acb317ed11a1a4a4daf67cc766df1471b9a3c50: Status 404 returned error can't find the container with id 3545637cdbb817797ebdd7286acb317ed11a1a4a4daf67cc766df1471b9a3c50 Oct 30 11:54:00.264999 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.264974 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f2f3c38-f710-491a-9728-7c641291d679-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:00.265139 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.265104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:00.265139 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:00.265120 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 11:54:00.265211 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:00.265177 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4f2f3c38-f710-491a-9728-7c641291d679-kube-rbac-proxy-cm podName:4f2f3c38-f710-491a-9728-7c641291d679 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:01.265160351 +0000 UTC m=+33.125379325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4f2f3c38-f710-491a-9728-7c641291d679-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6dvql" (UID: "4f2f3c38-f710-491a-9728-7c641291d679") : configmap "kube-rbac-proxy" not found Oct 30 11:54:00.265211 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:00.265190 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:00.265281 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:00.265227 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls podName:4f2f3c38-f710-491a-9728-7c641291d679 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:01.265216341 +0000 UTC m=+33.125435326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dvql" (UID: "4f2f3c38-f710-491a-9728-7c641291d679") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:00.365797 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.365757 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/778f7f74-6134-47ab-a978-cf3462316e72-signing-key\") pod \"service-ca-5fffc97bdf-gqgxk\" (UID: \"778f7f74-6134-47ab-a978-cf3462316e72\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" Oct 30 11:54:00.365986 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.365824 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/778f7f74-6134-47ab-a978-cf3462316e72-signing-cabundle\") pod \"service-ca-5fffc97bdf-gqgxk\" (UID: \"778f7f74-6134-47ab-a978-cf3462316e72\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" Oct 30 11:54:00.365986 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.365877 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpwfh\" (UniqueName: \"kubernetes.io/projected/778f7f74-6134-47ab-a978-cf3462316e72-kube-api-access-xpwfh\") pod \"service-ca-5fffc97bdf-gqgxk\" (UID: \"778f7f74-6134-47ab-a978-cf3462316e72\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" Oct 30 11:54:00.466713 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.466669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/778f7f74-6134-47ab-a978-cf3462316e72-signing-key\") pod \"service-ca-5fffc97bdf-gqgxk\" (UID: \"778f7f74-6134-47ab-a978-cf3462316e72\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" Oct 30 11:54:00.466903 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.466736 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/778f7f74-6134-47ab-a978-cf3462316e72-signing-cabundle\") pod \"service-ca-5fffc97bdf-gqgxk\" (UID: \"778f7f74-6134-47ab-a978-cf3462316e72\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" Oct 30 11:54:00.466903 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.466762 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpwfh\" (UniqueName: \"kubernetes.io/projected/778f7f74-6134-47ab-a978-cf3462316e72-kube-api-access-xpwfh\") pod \"service-ca-5fffc97bdf-gqgxk\" (UID: \"778f7f74-6134-47ab-a978-cf3462316e72\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" Oct 30 11:54:00.467373 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.467351 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/778f7f74-6134-47ab-a978-cf3462316e72-signing-cabundle\") pod \"service-ca-5fffc97bdf-gqgxk\" (UID: \"778f7f74-6134-47ab-a978-cf3462316e72\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" Oct 30 11:54:00.469123 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.469103 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/778f7f74-6134-47ab-a978-cf3462316e72-signing-key\") pod \"service-ca-5fffc97bdf-gqgxk\" (UID: \"778f7f74-6134-47ab-a978-cf3462316e72\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" Oct 30 11:54:00.490881 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.490852 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpwfh\" (UniqueName: \"kubernetes.io/projected/778f7f74-6134-47ab-a978-cf3462316e72-kube-api-access-xpwfh\") pod \"service-ca-5fffc97bdf-gqgxk\" (UID: \"778f7f74-6134-47ab-a978-cf3462316e72\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" Oct 30 11:54:00.546971 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.546927 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" Oct 30 11:54:00.583219 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.583193 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:54:00.583219 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.583235 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:54:00.583439 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.583257 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:54:00.600768 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.599853 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 11:54:00.600768 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.600183 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 11:54:00.600768 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.600448 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jjcjh\"" Oct 30 11:54:00.600768 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.600509 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 11:54:00.600768 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.600556 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rpjw7\"" Oct 30 11:54:00.600768 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.600592 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 11:54:00.600768 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.600603 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zfblq\"" Oct 30 11:54:00.862910 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.862877 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-gqgxk"] Oct 30 11:54:00.889562 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.889080 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z6hh6" event={"ID":"9d23976a-d44f-4b71-8b58-04932007cc17","Type":"ContainerStarted","Data":"8924ad7e7609d36b9309221ed6f63cf95f9beb316abade30a75232ef6b4f1000"} Oct 30 11:54:00.890863 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.890834 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" event={"ID":"778f7f74-6134-47ab-a978-cf3462316e72","Type":"ContainerStarted","Data":"daca8037d7968a526b51f86312be7352a9b2919bc73624fb7f50ff1f1882aed9"} Oct 30 11:54:00.892369 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:00.892343 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nr74p-m5v7z" event={"ID":"cf0f7768-5ba5-4b50-a601-0c4202b2b66e","Type":"ContainerStarted","Data":"3545637cdbb817797ebdd7286acb317ed11a1a4a4daf67cc766df1471b9a3c50"} Oct 30 11:54:01.174419 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:01.174380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:54:01.174605 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:01.174562 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:54:01.174669 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:01.174637 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls podName:1a948657-c616-435c-90db-b880a0db5bd9 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:03.174620326 +0000 UTC m=+35.034839328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls") pod "dns-default-p4nvb" (UID: "1a948657-c616-435c-90db-b880a0db5bd9") : secret "dns-default-metrics-tls" not found Oct 30 11:54:01.275170 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:01.275117 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:01.275369 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:01.275189 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f2f3c38-f710-491a-9728-7c641291d679-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:01.275369 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:01.275311 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:01.275446 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:01.275389 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls podName:4f2f3c38-f710-491a-9728-7c641291d679 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:03.275369661 +0000 UTC m=+35.135588648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dvql" (UID: "4f2f3c38-f710-491a-9728-7c641291d679") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:01.275681 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:01.275312 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 11:54:01.275809 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:01.275718 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4f2f3c38-f710-491a-9728-7c641291d679-kube-rbac-proxy-cm podName:4f2f3c38-f710-491a-9728-7c641291d679 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:03.275702325 +0000 UTC m=+35.135921313 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4f2f3c38-f710-491a-9728-7c641291d679-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6dvql" (UID: "4f2f3c38-f710-491a-9728-7c641291d679") : configmap "kube-rbac-proxy" not found Oct 30 11:54:02.487936 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:02.487889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs\") pod \"network-metrics-daemon-rh7rw\" (UID: \"260d2659-42a8-461f-8994-0c5f9522a80d\") " pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:54:02.488439 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:02.487954 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jzbb\" (UniqueName: \"kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb\") pod \"network-check-target-ld8pr\" (UID: \"363a0adc-193f-48d3-9628-1b9d10665597\") " pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:54:02.488439 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:02.488006 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:54:02.488439 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:02.488131 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 11:54:02.488439 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:02.488210 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs podName:260d2659-42a8-461f-8994-0c5f9522a80d nodeName:}" failed. No retries permitted until 2026-10-30 11:54:34.488188119 +0000 UTC m=+66.348407109 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs") pod "network-metrics-daemon-rh7rw" (UID: "260d2659-42a8-461f-8994-0c5f9522a80d") : secret "metrics-daemon-secret" not found Oct 30 11:54:02.490877 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:02.490855 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:54:02.491328 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:02.491303 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jzbb\" (UniqueName: \"kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb\") pod \"network-check-target-ld8pr\" (UID: \"363a0adc-193f-48d3-9628-1b9d10665597\") " pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:54:02.763624 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:02.763601 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:54:02.780371 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:02.780341 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:54:02.900114 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:02.900059 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nr74p-m5v7z" event={"ID":"cf0f7768-5ba5-4b50-a601-0c4202b2b66e","Type":"ContainerStarted","Data":"4e44951acbbd226653848fc81fd977d4c9e6d45583cef8d14e912a75d4f07d02"} Oct 30 11:54:02.946385 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:02.946043 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-nr74p-m5v7z" podStartSLOduration=1.441608601 podStartE2EDuration="3.946012806s" totalImagesPullingTime="2.504404205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:02.94509375 +0000 UTC m=+34.805312739" watchObservedRunningTime="2026-10-30 11:54:02.946012806 +0000 UTC m=+34.806231801" Oct 30 11:54:03.075939 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:03.075911 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mqv9x"] Oct 30 11:54:03.169883 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:03.169856 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ld8pr"] Oct 30 11:54:03.192855 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:03.192818 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:54:03.193011 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:03.192993 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:54:03.193061 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:03.193052 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls podName:1a948657-c616-435c-90db-b880a0db5bd9 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:07.193038437 +0000 UTC m=+39.053257410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls") pod "dns-default-p4nvb" (UID: "1a948657-c616-435c-90db-b880a0db5bd9") : secret "dns-default-metrics-tls" not found Oct 30 11:54:03.215512 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:54:03.215474 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46289f03_406c_4831_9ad1_0e6751fd7bd5.slice/crio-ecca7198a975105584690498b9ce1123acee1a0c6317cc908f4f27982599bcc2 WatchSource:0}: Error finding container ecca7198a975105584690498b9ce1123acee1a0c6317cc908f4f27982599bcc2: Status 404 returned error can't find the container with id ecca7198a975105584690498b9ce1123acee1a0c6317cc908f4f27982599bcc2 Oct 30 11:54:03.215918 ip-10-0-131-76 kubenswrapper[2574]: W1030 11:54:03.215888 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363a0adc_193f_48d3_9628_1b9d10665597.slice/crio-3f42e28b337da0af37c779d0fbb1fd17de5d3be8af05390d241dd9e63739784f WatchSource:0}: Error finding container 3f42e28b337da0af37c779d0fbb1fd17de5d3be8af05390d241dd9e63739784f: Status 404 returned error can't find the container with id 3f42e28b337da0af37c779d0fbb1fd17de5d3be8af05390d241dd9e63739784f Oct 30 11:54:03.293731 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:03.293696 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:03.293883 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:03.293739 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f2f3c38-f710-491a-9728-7c641291d679-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:03.293883 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:03.293829 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 11:54:03.293883 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:03.293861 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:03.293883 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:03.293880 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4f2f3c38-f710-491a-9728-7c641291d679-kube-rbac-proxy-cm podName:4f2f3c38-f710-491a-9728-7c641291d679 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:07.293867346 +0000 UTC m=+39.154086320 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4f2f3c38-f710-491a-9728-7c641291d679-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6dvql" (UID: "4f2f3c38-f710-491a-9728-7c641291d679") : configmap "kube-rbac-proxy" not found Oct 30 11:54:03.294041 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:03.293926 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls podName:4f2f3c38-f710-491a-9728-7c641291d679 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:07.293906683 +0000 UTC m=+39.154125660 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dvql" (UID: "4f2f3c38-f710-491a-9728-7c641291d679") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:03.904841 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:03.904768 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mqv9x" event={"ID":"46289f03-406c-4831-9ad1-0e6751fd7bd5","Type":"ContainerStarted","Data":"ecca7198a975105584690498b9ce1123acee1a0c6317cc908f4f27982599bcc2"} Oct 30 11:54:03.906806 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:03.906778 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" event={"ID":"778f7f74-6134-47ab-a978-cf3462316e72","Type":"ContainerStarted","Data":"9f0fc7599b2844c26a847befec520e37872677b748196efccf90edec385ac874"} Oct 30 11:54:03.908126 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:03.908100 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ld8pr" event={"ID":"363a0adc-193f-48d3-9628-1b9d10665597","Type":"ContainerStarted","Data":"3f42e28b337da0af37c779d0fbb1fd17de5d3be8af05390d241dd9e63739784f"} Oct 30 11:54:03.909872 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:03.909849 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z6hh6" event={"ID":"9d23976a-d44f-4b71-8b58-04932007cc17","Type":"ContainerStarted","Data":"0db13a87c5c82e1e1f9bcba4f7b8c32a7e0da1ef05ced8ebcdc5b34b2dea9456"} Oct 30 11:54:03.951983 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:03.951921 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" podStartSLOduration=1.069760583 podStartE2EDuration="3.951901815s" totalImagesPullingTime="2.882141232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:03.950796338 +0000 UTC m=+35.811015334" watchObservedRunningTime="2026-10-30 11:54:03.951901815 +0000 UTC m=+35.812120811" Oct 30 11:54:03.994298 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:03.994250 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-z6hh6" podStartSLOduration=1.880491804 podStartE2EDuration="4.994230745s" totalImagesPullingTime="3.113738941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:03.994014938 +0000 UTC m=+35.854233932" watchObservedRunningTime="2026-10-30 11:54:03.994230745 +0000 UTC m=+35.854449741" Oct 30 11:54:06.065687 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:06.065658 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nhbdl_617427c7-590d-4d80-8c4e-e11c7812dca0/dns-node-resolver/0.log" Oct 30 11:54:06.866434 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:06.866405 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7c297_29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e/node-ca/0.log" Oct 30 11:54:07.138616 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:07.138535 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qjvts_f15ba445-e38a-46f6-be0f-65f7307e79d6/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:07.227964 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:07.227929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:54:07.228147 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:07.228089 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:54:07.228214 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:07.228165 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls podName:1a948657-c616-435c-90db-b880a0db5bd9 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:15.22814506 +0000 UTC m=+47.088364038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls") pod "dns-default-p4nvb" (UID: "1a948657-c616-435c-90db-b880a0db5bd9") : secret "dns-default-metrics-tls" not found Oct 30 11:54:07.274001 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:07.273964 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qjvts_f15ba445-e38a-46f6-be0f-65f7307e79d6/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:07.328811 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:07.328779 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:07.328927 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:07.328832 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f2f3c38-f710-491a-9728-7c641291d679-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:07.328984 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:07.328946 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:07.329039 ip-10-0-131-76 kubenswrapper[2574]: E1030 11:54:07.329027 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls podName:4f2f3c38-f710-491a-9728-7c641291d679 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:15.329006932 +0000 UTC m=+47.189225918 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dvql" (UID: "4f2f3c38-f710-491a-9728-7c641291d679") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:07.329368 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:07.329348 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f2f3c38-f710-491a-9728-7c641291d679-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:07.665032 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:07.664999 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qjvts_f15ba445-e38a-46f6-be0f-65f7307e79d6/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:07.859558 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:07.859516 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qjvts_f15ba445-e38a-46f6-be0f-65f7307e79d6/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:07.922546 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:07.922465 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ld8pr" event={"ID":"363a0adc-193f-48d3-9628-1b9d10665597","Type":"ContainerStarted","Data":"a4ff1979f93bb5bee021dd71ab613c2f1ddbb7f57bb10bb577430d0ff93149e4"} Oct 30 11:54:07.922688 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:07.922666 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:54:07.965610 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:07.965553 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ld8pr" podStartSLOduration=34.955303369 podStartE2EDuration="38.965537914s" totalImagesPullingTime="4.010234545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:07.964464126 +0000 UTC m=+39.824683123" watchObservedRunningTime="2026-10-30 11:54:07.965537914 +0000 UTC m=+39.825756909" Oct 30 11:54:08.066322 ip-10-0-131-76 kubenswrapper[2574]: I1030 11:54:08.066292 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z6hh6_9d23976a-d44f-4b71-8b58-04932007cc17/serve-healthcheck-canary/0.log" Oct 30 11:54:08.874210 ip-10-0-131-76 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 11:54:08.885450 ip-10-0-131-76 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 11:54:08.885617 ip-10-0-131-76 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 11:54:08.885666 ip-10-0-131-76 systemd[1]: kubelet.service: Consumed 2.427s CPU time. Oct 30 11:54:08.900761 ip-10-0-131-76 systemd[1]: Starting Kubernetes Kubelet... Oct 30 11:54:09.004961 ip-10-0-131-76 kubenswrapper[4468]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:54:09.004961 ip-10-0-131-76 kubenswrapper[4468]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 11:54:09.004961 ip-10-0-131-76 kubenswrapper[4468]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:54:09.004961 ip-10-0-131-76 kubenswrapper[4468]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:54:09.007219 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007131 4468 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:09.007219 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007219 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:09.007219 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007223 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:09.007219 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007227 4468 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:09.007348 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007230 4468 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:09.007348 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007233 4468 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:09.007348 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007236 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:09.007348 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007239 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:09.007348 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007241 4468 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:09.007348 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007244 4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:09.007348 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007247 4468 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:09.007348 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007250 4468 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:09.007348 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007252 4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:09.007348 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007255 4468 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:09.007348 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007258 4468 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:09.007348 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007261 4468 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:09.007348 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007264 4468 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:09.007348 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007266 4468 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:09.007348 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007269 4468 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:09.007348 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007271 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:09.007348 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007275 4468 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:09.007348 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007278 4468 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:09.007348 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007280 4468 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:09.007819 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007283 4468 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:09.007819 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007286 4468 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:09.007819 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007288 4468 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:09.007819 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007298 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:09.007819 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007301 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:09.007819 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007304 4468 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:09.007819 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007307 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:09.007819 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007310 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:09.007819 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007314 4468 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:09.007819 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007317 4468 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:09.007819 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007319 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:09.007819 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007322 4468 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:09.007819 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007324 4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:09.007819 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007327 4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:09.007819 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007331 4468 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:09.007819 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007335 4468 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:09.007819 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007339 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:09.007819 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007342 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:09.008307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007345 4468 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:09.008307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007349 4468 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:09.008307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007352 4468 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:09.008307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007354 4468 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:09.008307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007358 4468 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:09.008307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007361 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:09.008307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007364 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:09.008307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007366 4468 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:09.008307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007369 4468 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:09.008307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007372 4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:09.008307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007374 4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:09.008307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007377 4468 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:09.008307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007380 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:09.008307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007383 4468 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:09.008307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007386 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:09.008307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007389 4468 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:09.008307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007392 4468 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:09.008307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007395 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:09.008307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007400 4468 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007404 4468 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007407 4468 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007409 4468 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007412 4468 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007415 4468 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007418 4468 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007421 4468 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007423 4468 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007426 4468 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007428 4468 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007431 4468 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007434 4468 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007436 4468 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007439 4468 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007442 4468 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007444 4468 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007447 4468 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007449 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007452 4468 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:09.008787 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007455 4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007457 4468 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007460 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007462 4468 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007465 4468 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007488 4468 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007493 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007496 4468 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007500 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007504 4468 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007507 4468 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007511 4468 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007514 4468 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007516 4468 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007519 4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007522 4468 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007525 4468 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007527 4468 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007530 4468 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007533 4468 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:09.009297 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007535 4468 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:09.009803 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007538 4468 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:09.009803 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007541 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:09.009803 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007543 4468 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:09.009803 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007546 4468 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:09.009803 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007549 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:09.009803 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.007552 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:09.009803 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008361 4468 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:09.009803 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008370 4468 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:09.009803 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008374 4468 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:09.009803 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008377 4468 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:09.009803 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008380 4468 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:09.009803 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008384 4468 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:09.009803 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008386 4468 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:09.009803 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008389 4468 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:09.009803 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008392 4468 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:09.009803 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008395 4468 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:09.009803 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008398 4468 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:09.009803 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008400 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:09.009803 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008404 4468 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:09.010304 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008408 4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:09.010304 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008413 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:09.010304 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008416 4468 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:09.010304 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008418 4468 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:09.010304 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008421 4468 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:09.010304 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008424 4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:09.010304 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008427 4468 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:09.010304 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008429 4468 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:09.010304 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008432 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:09.010304 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008435 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:09.010304 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008438 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:09.010304 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008441 4468 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:09.010304 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008443 4468 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:09.010304 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008446 4468 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:09.010304 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008448 4468 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:09.010304 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008451 4468 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:09.010304 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008453 4468 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:09.010304 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008456 4468 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:09.010304 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008459 4468 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:09.010776 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008461 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:09.010776 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008464 4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:09.010776 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008468 4468 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:09.010776 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008471 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:09.010776 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008475 4468 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:09.010776 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008477 4468 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:09.010776 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008480 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:09.010776 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008483 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:09.010776 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008486 4468 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:09.010776 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008488 4468 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:09.010776 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008491 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:09.010776 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008493 4468 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:09.010776 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008496 4468 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:09.010776 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008499 4468 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:09.010776 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008502 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:09.010776 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008504 4468 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:09.010776 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008507 4468 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:09.010776 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008510 4468 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:09.010776 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008513 4468 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008515 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008518 4468 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008521 4468 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008523 4468 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008526 4468 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008528 4468 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008531 4468 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008533 4468 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008536 4468 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008538 4468 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008541 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008543 4468 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008546 4468 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008548 4468 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008551 4468 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008554 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008556 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008559 4468 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008562 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:09.011258 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008564 4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008567 4468 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008569 4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008572 4468 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008574 4468 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008577 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008582 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008586 4468 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008588 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008591 4468 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008593 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008596 4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008598 4468 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008601 4468 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008604 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008606 4468 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008609 4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008611 4468 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008615 4468 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008618 4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:09.011767 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008620 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008623 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008625 4468 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008628 4468 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008630 4468 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008633 4468 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008635 4468 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008638 4468 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008640 4468 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008643 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008645 4468 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008647 4468 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008650 4468 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008652 4468 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008656 4468 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.008658 4468 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009203 4468 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009210 4468 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009217 4468 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009220 4468 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:09.012307 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009222 4468 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009225 4468 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009228 4468 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009231 4468 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009234 4468 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009236 4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009239 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009242 4468 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009244 4468 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009247 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009250 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009253 4468 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009255 4468 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009258 4468 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009260 4468 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009263 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009265 4468 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009268 4468 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009271 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009274 4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:09.012799 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009276 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009278 4468 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009281 4468 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009284 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009286 4468 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009289 4468 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009292 4468 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009294 4468 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009297 4468 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009300 4468 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009304 4468 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009307 4468 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009310 4468 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009312 4468 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009315 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009317 4468 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009320 4468 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009323 4468 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009325 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009328 4468 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:09.013345 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009330 4468 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009333 4468 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009335 4468 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009338 4468 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009341 4468 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009343 4468 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009346 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009348 4468 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009351 4468 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009353 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009356 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009359 4468 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009361 4468 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009365 4468 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009369 4468 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009372 4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009375 4468 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009378 4468 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009381 4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009385 4468 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:09.013879 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009388 4468 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:09.014409 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009391 4468 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:09.014409 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009394 4468 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:09.014409 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009397 4468 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:09.014409 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009399 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:09.014409 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009402 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:09.014409 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009404 4468 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:09.014409 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009407 4468 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:09.014409 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009409 4468 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:09.014409 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009412 4468 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:09.014409 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009415 4468 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:09.014409 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009417 4468 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:09.014409 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009419 4468 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:09.014409 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009423 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:09.014409 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009426 4468 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:09.014409 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009428 4468 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:09.014409 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009431 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:09.014409 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009433 4468 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:09.014409 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009436 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:09.014409 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009449 4468 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:09.014866 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009453 4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:09.014866 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009456 4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:09.014866 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009459 4468 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:09.014866 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009461 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:09.014866 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009464 4468 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:09.014866 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009466 4468 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:09.014866 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009469 4468 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:09.014866 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009471 4468 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:09.014866 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009474 4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:09.014866 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009476 4468 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:09.014866 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009479 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:09.014866 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009483 4468 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:09.014866 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009486 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:09.014866 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009488 4468 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:09.014866 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009491 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:09.014866 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009494 4468 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:09.014866 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009496 4468 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:09.014866 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009501 4468 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:09.014866 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009504 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009507 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009510 4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.009514 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009603 4468 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009614 4468 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009621 4468 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009625 4468 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009630 4468 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009634 4468 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009639 4468 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009644 4468 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009647 4468 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009651 4468 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009654 4468 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009658 4468 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009661 4468 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009664 4468 flags.go:64] FLAG: --cgroup-root="" Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009667 4468 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009671 4468 flags.go:64] FLAG: --client-ca-file="" Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009674 4468 flags.go:64] FLAG: --cloud-provider="external" Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009677 4468 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009685 4468 flags.go:64] FLAG: --cluster-domain="" Oct 30 11:54:09.015334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009688 4468 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009691 4468 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009696 4468 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009700 4468 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009704 4468 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009708 4468 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009711 4468 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009714 4468 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009717 4468 flags.go:64] FLAG: --contention-profiling="false" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009720 4468 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009723 4468 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009726 4468 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009729 4468 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009733 4468 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009736 4468 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009739 4468 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009742 4468 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009745 4468 flags.go:64] FLAG: --enable-server="true" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009748 4468 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009752 4468 flags.go:64] FLAG: --event-burst="100" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009756 4468 flags.go:64] FLAG: --event-qps="50" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009758 4468 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009762 4468 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009765 4468 flags.go:64] FLAG: --eviction-hard="" Oct 30 11:54:09.015998 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009769 4468 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009772 4468 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009775 4468 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009778 4468 flags.go:64] FLAG: --eviction-soft="" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009782 4468 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009785 4468 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009787 4468 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009790 4468 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009793 4468 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009796 4468 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009800 4468 flags.go:64] FLAG: --feature-gates="" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009804 4468 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009808 4468 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009812 4468 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009816 4468 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009819 4468 flags.go:64] FLAG: --healthz-port="10248" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009822 4468 flags.go:64] FLAG: --help="false" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009826 4468 flags.go:64] FLAG: --hostname-override="ip-10-0-131-76.ec2.internal" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009829 4468 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009832 4468 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009835 4468 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009838 4468 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009841 4468 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009844 4468 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 11:54:09.016649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009847 4468 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009850 4468 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009853 4468 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009855 4468 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009858 4468 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009861 4468 flags.go:64] FLAG: --kube-reserved="" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009866 4468 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009869 4468 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009872 4468 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009875 4468 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009878 4468 flags.go:64] FLAG: --lock-file="" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009881 4468 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009884 4468 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009886 4468 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009892 4468 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009895 4468 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009898 4468 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009901 4468 flags.go:64] FLAG: --logging-format="text" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009905 4468 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009909 4468 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009912 4468 flags.go:64] FLAG: --manifest-url="" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009915 4468 flags.go:64] FLAG: --manifest-url-header="" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009923 4468 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009927 4468 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009931 4468 flags.go:64] FLAG: --max-pods="110" Oct 30 11:54:09.017251 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009934 4468 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009938 4468 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009941 4468 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009943 4468 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009946 4468 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009949 4468 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009952 4468 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009960 4468 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009963 4468 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009966 4468 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009969 4468 flags.go:64] FLAG: --pod-cidr="" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009972 4468 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009975 4468 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009980 4468 flags.go:64] FLAG: --pods-per-core="0" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009983 4468 flags.go:64] FLAG: --port="10250" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009986 4468 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009989 4468 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0541f4526f2866583" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009992 4468 flags.go:64] FLAG: --qos-reserved="" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009996 4468 flags.go:64] FLAG: --read-only-port="10255" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.009998 4468 flags.go:64] FLAG: --register-node="true" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010001 4468 flags.go:64] FLAG: --register-with-taints="" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010005 4468 flags.go:64] FLAG: --registry-burst="10" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010008 4468 flags.go:64] FLAG: --registry-qps="5" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010011 4468 flags.go:64] FLAG: --reserved-cpus="" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010014 4468 flags.go:64] FLAG: --reserved-memory="" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010019 4468 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 11:54:09.017997 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010023 4468 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010026 4468 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010029 4468 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010032 4468 flags.go:64] FLAG: --runonce="false" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010041 4468 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010044 4468 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010047 4468 flags.go:64] FLAG: --seccomp-default="false" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010051 4468 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010053 4468 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010056 4468 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010060 4468 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010063 4468 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010079 4468 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010084 4468 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010089 4468 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010093 4468 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010098 4468 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010102 4468 flags.go:64] FLAG: --system-cgroups="" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010107 4468 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010116 4468 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010119 4468 flags.go:64] FLAG: --tls-cert-file="" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010122 4468 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010127 4468 flags.go:64] FLAG: --tls-min-version="" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010130 4468 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010133 4468 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 11:54:09.018657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010136 4468 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010140 4468 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010143 4468 flags.go:64] FLAG: --v="2" Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010154 4468 flags.go:64] FLAG: --version="false" Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010158 4468 flags.go:64] FLAG: --vmodule="" Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010162 4468 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010167 4468 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010303 4468 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010307 4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010311 4468 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010314 4468 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010317 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010320 4468 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010323 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010326 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010328 4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010331 4468 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010333 4468 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010350 4468 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010354 4468 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010357 4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010360 4468 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:09.019461 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010364 4468 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:09.020306 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010367 4468 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:09.020306 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010370 4468 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:09.020306 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010373 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:09.020306 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010377 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:09.020306 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010380 4468 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:09.020306 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010382 4468 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:09.020306 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010385 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:09.020306 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010388 4468 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:09.020306 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010390 4468 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:09.020306 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010393 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:09.020306 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010396 4468 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:09.020306 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010398 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:09.020306 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010401 4468 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:09.020306 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010404 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:09.020306 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010406 4468 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:09.020306 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010410 4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:09.020306 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010413 4468 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:09.020306 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010416 4468 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:09.020306 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010418 4468 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:09.020891 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010421 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:09.020891 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010424 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:09.020891 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010426 4468 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:09.020891 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010429 4468 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:09.020891 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010432 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:09.020891 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010435 4468 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:09.020891 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010437 4468 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:09.020891 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010440 4468 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:09.020891 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010442 4468 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:09.020891 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010445 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:09.020891 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010447 4468 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:09.020891 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010450 4468 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:09.020891 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010452 4468 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:09.020891 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010455 4468 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:09.020891 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010457 4468 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:09.020891 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010460 4468 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:09.020891 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010464 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:09.020891 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010466 4468 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:09.020891 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010469 4468 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:09.021500 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010471 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:09.021500 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010474 4468 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:09.021500 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010477 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:09.021500 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010479 4468 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:09.021500 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010482 4468 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:09.021500 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010484 4468 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:09.021500 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010487 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:09.021500 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010490 4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:09.021500 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010492 4468 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:09.021500 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010497 4468 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:09.021500 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010500 4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:09.021500 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010502 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:09.021500 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010505 4468 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:09.021500 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010507 4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:09.021500 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010510 4468 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:09.021500 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010512 4468 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:09.021500 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010515 4468 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:09.021500 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010518 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:09.021500 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010520 4468 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010523 4468 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010527 4468 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010531 4468 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010534 4468 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010536 4468 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010539 4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010541 4468 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010544 4468 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010547 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010550 4468 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010554 4468 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010556 4468 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010559 4468 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010561 4468 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010564 4468 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010567 4468 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010569 4468 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010572 4468 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010574 4468 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:09.022037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010577 4468 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:09.022633 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010579 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:09.022633 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010582 4468 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:09.022633 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010586 4468 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:09.022633 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010588 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:09.022633 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010591 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:09.022633 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010593 4468 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:09.022633 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010596 4468 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:09.022633 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010599 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:09.022633 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010601 4468 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:09.022633 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010604 4468 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:09.022633 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010607 4468 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:09.022633 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010611 4468 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:09.022633 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010614 4468 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:09.022633 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.010639 4468 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:54:09.022633 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010676 4468 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:09.022633 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010681 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010684 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010687 4468 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010690 4468 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010692 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010695 4468 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010699 4468 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010701 4468 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010706 4468 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010710 4468 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010712 4468 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010715 4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010718 4468 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010720 4468 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010723 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010725 4468 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010728 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010730 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010733 4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010737 4468 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:09.023031 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010739 4468 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010742 4468 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010744 4468 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010747 4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010750 4468 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010752 4468 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010755 4468 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010757 4468 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010760 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010763 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010766 4468 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010768 4468 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010771 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010774 4468 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010776 4468 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010779 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010782 4468 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010784 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010787 4468 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010790 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:09.023756 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010793 4468 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:09.024616 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010796 4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:09.024616 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010799 4468 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:09.024616 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010801 4468 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:09.024616 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010804 4468 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:09.024616 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010807 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:09.024616 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010809 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:09.024616 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010812 4468 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:09.024616 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010814 4468 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:09.024616 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010817 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:09.024616 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010819 4468 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:09.024616 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010826 4468 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:09.024616 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010829 4468 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:09.024616 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010832 4468 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:09.024616 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010834 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:09.024616 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010837 4468 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:09.024616 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010839 4468 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:09.024616 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010842 4468 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:09.024616 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010844 4468 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:09.024616 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010847 4468 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:09.025135 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010850 4468 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:09.025135 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010852 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:09.025135 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010855 4468 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:09.025135 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010858 4468 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:09.025135 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010860 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:09.025135 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010863 4468 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:09.025135 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010866 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:09.025135 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010869 4468 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:09.025135 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010871 4468 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:09.025135 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010874 4468 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:09.025135 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010876 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:09.025135 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010879 4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:09.025135 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010883 4468 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:09.025135 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010885 4468 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:09.025135 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010888 4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:09.025135 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010891 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:09.025135 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010893 4468 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:09.025135 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010896 4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:09.025135 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010898 4468 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010901 4468 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010904 4468 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010906 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010909 4468 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010913 4468 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010917 4468 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010921 4468 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010923 4468 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010926 4468 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010929 4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010931 4468 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010933 4468 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010936 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010940 4468 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010942 4468 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010945 4468 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010948 4468 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010950 4468 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010953 4468 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:09.025602 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010955 4468 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010958 4468 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010960 4468 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010963 4468 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010965 4468 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010968 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.010972 4468 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.014777 4468 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.014795 4468 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.014811 4468 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.014815 4468 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014871 4468 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014876 4468 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014880 4468 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014884 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014887 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014889 4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014892 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014895 4468 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014897 4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:09.026102 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014900 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:09.026610 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014903 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:09.026610 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014905 4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:09.026610 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014908 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:09.026610 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014911 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:09.026610 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014914 4468 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:09.026610 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014917 4468 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:09.026610 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014920 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:09.026610 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014922 4468 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:09.026610 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014925 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:09.026610 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014928 4468 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:09.026610 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014930 4468 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:09.026610 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014933 4468 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:09.026610 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014935 4468 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:09.026610 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014938 4468 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:09.026610 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014940 4468 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:09.026610 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014945 4468 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:09.026610 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014949 4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:09.026610 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014952 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014955 4468 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014959 4468 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014963 4468 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014966 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014969 4468 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014972 4468 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014975 4468 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014977 4468 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014980 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014983 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014985 4468 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014988 4468 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014991 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014993 4468 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014996 4468 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.014998 4468 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015001 4468 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015004 4468 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015006 4468 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:09.027036 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015009 4468 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015012 4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015014 4468 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015017 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015019 4468 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015022 4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015024 4468 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015027 4468 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015029 4468 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015032 4468 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015035 4468 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015037 4468 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015040 4468 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015043 4468 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015046 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015049 4468 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015052 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015055 4468 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015057 4468 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015060 4468 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:09.027589 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015062 4468 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015080 4468 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015084 4468 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015087 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015090 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015093 4468 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015095 4468 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015098 4468 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015101 4468 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015103 4468 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015106 4468 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015109 4468 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015111 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015114 4468 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015117 4468 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015120 4468 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015123 4468 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015125 4468 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015128 4468 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015130 4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:09.028100 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015133 4468 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:09.028586 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015135 4468 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:09.028586 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015138 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:09.028586 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015141 4468 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:09.028586 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015143 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:09.028586 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015146 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:09.028586 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015148 4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:09.028586 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015151 4468 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:09.028586 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015154 4468 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:09.028586 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015157 4468 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:09.028586 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015160 4468 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:09.028586 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015162 4468 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:09.028586 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015165 4468 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:09.028586 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015167 4468 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:09.028586 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015170 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:09.028586 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015173 4468 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:09.028586 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015177 4468 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:09.028586 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015181 4468 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:09.028586 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015184 4468 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:09.029037 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.015212 4468 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:54:09.029037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015369 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:09.029037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015374 4468 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:09.029037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015377 4468 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:09.029037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015380 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:09.029037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015383 4468 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:09.029037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015386 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:09.029037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015389 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:09.029037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015391 4468 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:09.029037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015393 4468 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:09.029037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015396 4468 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:09.029037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015399 4468 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:09.029037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015401 4468 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:09.029037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015404 4468 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:09.029037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015406 4468 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:09.029037 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015409 4468 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:09.029521 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015413 4468 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:09.029521 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015417 4468 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:09.029521 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015420 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:09.029521 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015422 4468 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:09.029521 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015425 4468 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:09.029521 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015428 4468 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:09.029521 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015431 4468 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:09.029521 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015434 4468 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:09.029521 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015438 4468 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:09.029521 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015441 4468 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:09.029521 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015444 4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:09.029521 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015447 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:09.029521 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015449 4468 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:09.029521 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015452 4468 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:09.029521 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015454 4468 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:09.029521 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015457 4468 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:09.029521 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015460 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:09.029521 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015462 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:09.029521 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015465 4468 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015468 4468 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015470 4468 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015473 4468 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015475 4468 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015478 4468 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015480 4468 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015483 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015485 4468 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015488 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015490 4468 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015493 4468 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015495 4468 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015498 4468 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015500 4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015503 4468 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015505 4468 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015508 4468 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015511 4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015513 4468 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:09.029996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015516 4468 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:09.030485 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015519 4468 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:09.030485 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015521 4468 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:09.030485 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015524 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:09.030485 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015526 4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:09.030485 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015529 4468 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:09.030485 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015531 4468 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:09.030485 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015534 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:09.030485 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015536 4468 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:09.030485 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015539 4468 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:09.030485 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015541 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:09.030485 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015544 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:09.030485 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015546 4468 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:09.030485 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015549 4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:09.030485 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015551 4468 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:09.030485 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015554 4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:09.030485 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015556 4468 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:09.030485 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015559 4468 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:09.030485 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015561 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:09.030485 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015564 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:09.030941 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015566 4468 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:09.030941 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015569 4468 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:09.030941 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015572 4468 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:09.030941 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015574 4468 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:09.030941 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015577 4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:09.030941 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015580 4468 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:09.030941 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015585 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:09.030941 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015587 4468 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:09.030941 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015590 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:09.030941 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015593 4468 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:09.030941 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015595 4468 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:09.030941 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015598 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:09.030941 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015600 4468 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:09.030941 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015603 4468 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:09.030941 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015606 4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:09.030941 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015608 4468 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:09.030941 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015611 4468 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:09.030941 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015614 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:09.030941 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015616 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:09.031447 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015619 4468 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:09.031447 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015622 4468 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:09.031447 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015624 4468 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:09.031447 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015627 4468 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:09.031447 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015629 4468 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:09.031447 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015632 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:09.031447 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015634 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:09.031447 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015637 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:09.031447 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015639 4468 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:09.031447 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015642 4468 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:09.031447 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015644 4468 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:09.031447 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015647 4468 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:09.031447 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015649 4468 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:09.031447 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015651 4468 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:09.031447 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.015674 4468 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:54:09.031447 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015704 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015707 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015710 4468 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015713 4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015717 4468 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015720 4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015723 4468 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015725 4468 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015728 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015731 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015733 4468 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015736 4468 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015739 4468 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015742 4468 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015744 4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015747 4468 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015750 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015752 4468 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015755 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015757 4468 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:09.031825 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015760 4468 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015763 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015765 4468 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015768 4468 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015770 4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015773 4468 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015775 4468 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015778 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015780 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015783 4468 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015785 4468 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015788 4468 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015790 4468 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015793 4468 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015795 4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015798 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015802 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015805 4468 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015807 4468 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015810 4468 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:09.032317 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015812 4468 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:09.032848 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015815 4468 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:09.032848 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015817 4468 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:09.032848 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015820 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:09.032848 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015823 4468 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:09.032848 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015825 4468 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:09.032848 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015828 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:09.032848 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015831 4468 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:09.032848 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015833 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:09.032848 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015836 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:09.032848 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015838 4468 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:09.032848 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015841 4468 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:09.032848 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015843 4468 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:09.032848 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015846 4468 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:09.032848 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015848 4468 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:09.032848 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015851 4468 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:09.032848 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015854 4468 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:09.032848 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015856 4468 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:09.032848 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015860 4468 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:09.032848 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015864 4468 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:09.033355 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015867 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:09.033355 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015870 4468 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:09.033355 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015872 4468 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:09.033355 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015875 4468 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:09.033355 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015877 4468 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:09.033355 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015881 4468 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:09.033355 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015884 4468 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:09.033355 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015887 4468 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:09.033355 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015892 4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:09.033355 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015895 4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:09.033355 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015898 4468 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:09.033355 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015901 4468 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:09.033355 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015903 4468 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:09.033355 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015906 4468 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:09.033355 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015908 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:09.033355 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015911 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:09.033355 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015913 4468 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:09.033355 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015916 4468 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:09.033355 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015918 4468 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015921 4468 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015937 4468 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015941 4468 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015944 4468 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015947 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015950 4468 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015953 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015956 4468 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015958 4468 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015961 4468 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015963 4468 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015966 4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015968 4468 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015971 4468 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015973 4468 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015976 4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015978 4468 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015980 4468 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015983 4468 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:09.033904 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015986 4468 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:09.034435 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015988 4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:09.034435 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015994 4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:09.034435 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015996 4468 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:09.034435 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.015999 4468 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:09.034435 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.016002 4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:09.034435 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.016004 4468 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:09.034435 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:09.016007 4468 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:09.034435 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.016059 4468 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 11:54:09.034435 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.017007 4468 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 11:54:09.034435 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.017055 4468 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 11:54:09.034435 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.017269 4468 server.go:1014] "Starting client certificate rotation" Oct 30 11:54:09.034435 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.017422 4468 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 11:54:09.034435 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.017497 4468 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 11:48:29 +0000 UTC" deadline="2028-07-06 00:10:50.961411863 +0000 UTC" Oct 30 11:54:09.034435 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.017537 4468 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14748h16m41.943877944s" Oct 30 11:54:09.034435 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.017753 4468 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 11:54:09.034435 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.018831 4468 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 11:54:09.034880 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.021707 4468 log.go:25] "Validated CRI v1 runtime API" Oct 30 11:54:09.034880 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.023293 4468 log.go:25] "Validated CRI v1 image API" Oct 30 11:54:09.034880 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.024022 4468 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 11:54:09.034880 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.025603 4468 fs.go:135] Filesystem UUIDs: map[088d176a-8d8a-4d6d-9485-cb6c6e501c0c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 dcb41eed-532c-473d-9714-19b7d1a2818b:/dev/nvme0n1p3] Oct 30 11:54:09.035094 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.025626 4468 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b6e449a33b18a7662a942a7cf450cb9b06ddb9743afb5dc310f3422a7cd2dab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b6e449a33b18a7662a942a7cf450cb9b06ddb9743afb5dc310f3422a7cd2dab/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3545637cdbb817797ebdd7286acb317ed11a1a4a4daf67cc766df1471b9a3c50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3545637cdbb817797ebdd7286acb317ed11a1a4a4daf67cc766df1471b9a3c50/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f42e28b337da0af37c779d0fbb1fd17de5d3be8af05390d241dd9e63739784f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f42e28b337da0af37c779d0fbb1fd17de5d3be8af05390d241dd9e63739784f/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4028eaff961969401c3554706f9bf3f13c181ffa0560bcbc20040c0aef366984/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4028eaff961969401c3554706f9bf3f13c181ffa0560bcbc20040c0aef366984/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/54b4e7b90dcf8224029f72fe0f07d265dbfcf68e7392a34072a420cc3cb09f76/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/54b4e7b90dcf8224029f72fe0f07d265dbfcf68e7392a34072a420cc3cb09f76/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8924ad7e7609d36b9309221ed6f63cf95f9beb316abade30a75232ef6b4f1000/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8924ad7e7609d36b9309221ed6f63cf95f9beb316abade30a75232ef6b4f1000/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5a46abc1f84e0d14e43fe38f08c99ad08b17614d7cf8be714cd793b499ded31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5a46abc1f84e0d14e43fe38f08c99ad08b17614d7cf8be714cd793b499ded31/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba041f17dddd5f9cd8954733ec48b40c3b715d59f841df308916d20f722ca28d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba041f17dddd5f9cd8954733ec48b40c3b715d59f841df308916d20f722ca28d/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5c242fec739eaa6bd477b4dbd05b2e5f8b9a8541dd712b1fae043d13dea8674/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5c242fec739eaa6bd477b4dbd05b2e5f8b9a8541dd712b1fae043d13dea8674/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd1ee92b2bc84070ab1e222d12ad2419f4a480741ac772a445913c7eb1e2543c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd1ee92b2bc84070ab1e222d12ad2419f4a480741ac772a445913c7eb1e2543c/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/daca8037d7968a526b51f86312be7352a9b2919bc73624fb7f50ff1f1882aed9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/daca8037d7968a526b51f86312be7352a9b2919bc73624fb7f50ff1f1882aed9/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e433332d61097a077363fdf206e7b57cbabb4a1b00b56772df19d032b73be9e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e433332d61097a077363fdf206e7b57cbabb4a1b00b56772df19d032b73be9e3/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ecc4d6e3358dd29e7f11102204c36b8bb652a103d91d031c431a2763ced53848/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ecc4d6e3358dd29e7f11102204c36b8bb652a103d91d031c431a2763ced53848/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ecca7198a975105584690498b9ce1123acee1a0c6317cc908f4f27982599bcc2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ecca7198a975105584690498b9ce1123acee1a0c6317cc908f4f27982599bcc2/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eee0a121e54d3c931a8ffcd5feed923fac046341f5f69d4af853f871c712bd1a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eee0a121e54d3c931a8ffcd5feed923fac046341f5f69d4af853f871c712bd1a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e/volumes/kubernetes.io~projected/kube-api-access-hj598:{mountpoint:/var/lib/kubelet/pods/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e/volumes/kubernetes.io~projected/kube-api-access-hj598 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a948657-c616-435c-90db-b880a0db5bd9/volumes/kubernetes.io~projected/kube-api-access-lmctv:{mountpoint:/var/lib/kubelet/pods/1a948657-c616-435c-90db-b880a0db5bd9/volumes/kubernetes.io~projected/kube-api-access-lmctv major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/260d2659-42a8-461f-8994-0c5f9522a80d/volumes/kubernetes.io~projected/kube-api-access-h8ccv:{mountpoint:/var/lib/kubelet/pods/260d2659-42a8-461f-8994-0c5f9522a80d/volumes/kubernetes.io~projected/kube-api-access-h8ccv major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e/volumes/kubernetes.io~projected/kube-api-access-l95t8:{mountpoint:/var/lib/kubelet/pods/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e/volumes/kubernetes.io~projected/kube-api-access-l95t8 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31df51a8-4f22-4745-b25f-8b69654f2845/volumes/kubernetes.io~projected/kube-api-access-v8tg9:{mountpoint:/var/lib/kubelet/pods/31df51a8-4f22-4745-b25f-8b69654f2845/volumes/kubernetes.io~projected/kube-api-access-v8tg9 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/363a0adc-193f-48d3-9628-1b9d10665597/volumes/kubernetes.io~projected/kube-api-access-2jzbb:{mountpoint:/var/lib/kubelet/pods/363a0adc-193f-48d3-9628-1b9d10665597/volumes/kubernetes.io~projected/kube-api-access-2jzbb major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46289f03-406c-4831-9ad1-0e6751fd7bd5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/46289f03-406c-4831-9ad1-0e6751fd7bd5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f2f3c38-f710-491a-9728-7c641291d679/volumes/kubernetes.io~projected/kube-api-access-nxpq6:{mountpoint:/var/lib/kubelet/pods/4f2f3c38-f710-491a-9728-7c641291d679/volumes/kubernetes.io~projected/kube-api-access-nxpq6 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/554f0b9b-bae8-478b-b3b1-85b376a79cc9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/554f0b9b-bae8-478b-b3b1-85b376a79cc9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/554f0b9b-bae8-478b-b3b1-85b376a79cc9/volumes/kubernetes.io~projected/kube-api-access-7jtv5:{mountpoint:/var/lib/kubelet/pods/554f0b9b-bae8-478b-b3b1-85b376a79cc9/volumes/kubernetes.io~projected/kube-api-access-7jtv5 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/554f0b9b-bae8-478b-b3b1-85b376a79cc9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/554f0b9b-bae8-478b-b3b1-85b376a79cc9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/617427c7-590d-4d80-8c4e-e11c7812dca0/volumes/kubernetes.io~projected/kube-api-access-5jhg2:{mountpoint:/var/lib/kubelet/pods/617427c7-590d-4d80-8c4e-e11c7812dca0/volumes/kubernetes.io~projected/kube-api-access-5jhg2 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/778f7f74-6134-47ab-a978-cf3462316e72/volumes/kubernetes.io~projected/kube-api-access-xpwfh:{mountpoint:/var/lib/kubelet/pods/778f7f74-6134-47ab-a978-cf3462316e72/volumes/kubernetes.io~projected/kube-api-access-xpwfh major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/778f7f74-6134-47ab-a978-cf3462316e72/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/778f7f74-6134-47ab-a978-cf3462316e72/volumes/kubernetes.io~secret/signing-key major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82a77175-bb28-474f-b4ec-dac73d790410/volumes/kubernetes.io~projected/kube-api-access-sp7n7:{mountpoint:/var/lib/kubelet/pods/82a77175-bb28-474f-b4ec-dac73d790410/volumes/kubernetes.io~projected/kube-api-access-sp7n7 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d23976a-d44f-4b71-8b58-04932007cc17/volumes/kubernetes.io~projected/kube-api-access-5p7nr:{mountpoint:/var/lib/kubelet/pods/9d23976a-d44f-4b71-8b58-04932007cc17/volumes/kubernetes.io~projected/kube-api-access-5p7nr major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d23976a-d44f-4b71-8b58-04932007cc17/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/9d23976a-d44f-4b71-8b58-04932007cc17/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2937779-b03d-4a5b-87bc-f062283eeb2e/volumes/kubernetes.io~projected/kube-api-access-bx8v8:{mountpoint:/var/lib/kubelet/pods/b2937779-b03d-4a5b-87bc-f062283eeb2e/volumes/kubernetes.io~projected/kube-api-access-bx8v8 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf0f7768-5ba5-4b50-a601-0c4202b2b66e/volumes/kubernetes.io~projected/kube-api-access-6lswd:{mountpoint:/var/lib/kubelet/pods/cf0f7768-5ba5-4b50-a601-0c4202b2b66e/volumes/kubernetes.io~projected/kube-api-access-6lswd major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d897418e-880a-46a9-8b13-9c2887bfddb7/volumes/kubernetes.io~projected/kube-api-access-s5g55:{mountpoint:/var/lib/kubelet/pods/d897418e-880a-46a9-8b13-9c2887bfddb7/volumes/kubernetes.io~projected/kube-api-access-s5g55 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f15ba445-e38a-46f6-be0f-65f7307e79d6/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/f15ba445-e38a-46f6-be0f-65f7307e79d6/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5403015fe127dfd3706fd3a744019a203fc96b0d2290959d7d36b5bc5ce47236/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/98914ec9c4397f9580160c06d49ee5caf72df629ad01f084cbf394fa5d450012/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/9225289a5b21acd7695e011155df2e8f8ff7bc55757f62e4d32f7237e85e759c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/1aee1e0f4ec987303d49dbe8f877fe77b5e748da013aa998a98829049043ef0f/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/aa201553f43d2267e2ba720d8135ad6940d9b253348680754f78d1287ed33d59/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c5153c7bdfa2c7539d139d7549e24e3186b20709b8bfe6c0687314f2e386c5c5/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f5f95291dade323d3db7387e6a5c06fd3eb0a27e277ae96d9f991c3738e706a2/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3c82d88f733ab74189e02a2732b55bda305266ab9703286cb5b6b275534dfce8/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/e0970210c46dc95c4fb882dc0bd7c9e72aaf57a3a48eb7b7da7444e41ce70eac/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/accdd1b944837305769b7f8c1711c651874600a2089fa88c1b463b5acadfd334/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/30c6c58067573ba73917d6157c51cf0b9a17a9f7c2a8315b64cbcbe5a227db82/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/f87e8f4fe733119dd8c3e377315372146cd1461a55698d1e39e45a92508c54e9/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/4736e2cb4c33472430dc51a1f73d0346c19e56b14e9870f144b6be8754a8dd2e/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9b5e9f3e9c167bc6f215e489f8d57caaec7b45e27aa82c4be1ec13ac762a428d/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/b5a6e9aeba13ad39d0db647e4f0bd6096e2acbc372f4299ffaba7da983b86846/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/45d1cf7b8bcc059a1f242fc41f43a10a16d3ccd1666b821599dca230d2d341d8/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/94ac328232cc9750ffddedd1acc23458020d8e30d2bccdc7dd5680ca609d54b7/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c86bbb2533bf258868000ecfea2bfae13c9b7e8bdf9d22cf374f98df7e3655c8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/2a2a55eb50b2ba3e9c5398c354adfa05b889763b28354b779d070088b0e5c3d5/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c00d140e8c2b8803fcaad8e84d830a3279c48a8ab67c40b73c3d78c54f20dca7/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e3b41ca29e4bcb1ae6fa7d57e13b8a0ed9c9d42f35a5c40c395e1764be499280/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/491e33eecb8ae67feeb94beb3c8bf7bdf53a13d8cc23ed14f299b8faf1829dfe/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/a9ea4fa077b33f475d407024743db1d8230030605b54d9e0e475c6f7dbf7d97c/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/ceae3d15b21783dcf899a33b29a847b5f7e560b80c4c75b0362fe7b7b26ceead/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/d48b7e37fee91936bca2b7d8d0a155e86d627e0ff1c9ab9a1bce282da34507ce/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/63bc28dbea0f9af144d935e40d2bdb42d780ee1860edae57707220221d158482/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/2ac504586b3f7c55c70e3f291e2b176ea9975225d7eec217da5509fc9879865a/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/ad340f2e301b9966ccaf4bd91583f3828107aa35746a113cea3fa5fc60bfa750/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/3a21bbb2f00e053fabe7619e75a3e460f097bcbdfbda7ecb3940cff14fae0245/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/bfc6fdb445fdac3148664d3143648d5f97a633b891793c7e8bc8ab9883cda6d7/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/4e0c299fbeef80a2e9d590db517ccf416759a52f199d8919b4cb44f0ef92d1ce/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/bd1eb2e047f99cf8b9ee675e23fcb6c72beb484115c7144fff03fa50e660a941/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/805aeeac148200c73d9a52a12c82c4eaa4164516182b49bf500a2edf0cb694ea/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/a652eb45e48c9220aa1f21e7f7172e47acee1e3073c63ad29695ef5ba57a5dbc/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/99852fb7a920fbccd47bfbff0e8b5ef23ff3bdda8e7907190ae9cfc80efe5138/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/7909b8e26480461604adc8bffd887665f9ce4be6b473f0be706b88d65a8df46b/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/096a69990a766cb892eefe7741e1b30ecdfd16511b5997606d70262683260118/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/0cae02ae301ba378d9c5f83463b8f1dabbdd4bd3d338db1ea7e448f005a8e91d/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/7bf6e2aa54a798a6b8b7cb1f6da4cccfa7debdbc82b756c03567a9a9c74e8f69/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/8ac7fbaf48f4aea567be55d52b0376e255a118c24895abfe59a7eadf7fd2bc99/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/c6e16ef0d6f6f82fbbb99724c504447e658d05cf090f0eec1d5b3c1c0c05af9c/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.032416 4468 manager.go:217] Machine: {Timestamp:2026-10-30 11:54:09.031953702 +0000 UTC m=+0.066092088 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100695 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d37fbb3ae1420407e84be2d27641d SystemUUID:ec2d37fb-b3ae-1420-407e-84be2d27641d BootID:938eaf26-5289-4369-a053-15d2ca83361a Filesystems:[{Device:/var/lib/kubelet/pods/554f0b9b-bae8-478b-b3b1-85b376a79cc9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5a46abc1f84e0d14e43fe38f08c99ad08b17614d7cf8be714cd793b499ded31/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d23976a-d44f-4b71-8b58-04932007cc17/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8924ad7e7609d36b9309221ed6f63cf95f9beb316abade30a75232ef6b4f1000/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ecc4d6e3358dd29e7f11102204c36b8bb652a103d91d031c431a2763ced53848/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba041f17dddd5f9cd8954733ec48b40c3b715d59f841df308916d20f722ca28d/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4f2f3c38-f710-491a-9728-7c641291d679/volumes/kubernetes.io~projected/kube-api-access-nxpq6 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3545637cdbb817797ebdd7286acb317ed11a1a4a4daf67cc766df1471b9a3c50/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f15ba445-e38a-46f6-be0f-65f7307e79d6/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d897418e-880a-46a9-8b13-9c2887bfddb7/volumes/kubernetes.io~projected/kube-api-access-s5g55 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82a77175-bb28-474f-b4ec-dac73d790410/volumes/kubernetes.io~projected/kube-api-access-sp7n7 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/daca8037d7968a526b51f86312be7352a9b2919bc73624fb7f50ff1f1882aed9/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/31df51a8-4f22-4745-b25f-8b69654f2845/volumes/kubernetes.io~projected/kube-api-access-v8tg9 DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf0f7768-5ba5-4b50-a601-0c4202b2b66e/volumes/kubernetes.io~projected/kube-api-access-6lswd DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eee0a121e54d3c931a8ffcd5feed923fac046341f5f69d4af853f871c712bd1a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/554f0b9b-bae8-478b-b3b1-85b376a79cc9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5c242fec739eaa6bd477b4dbd05b2e5f8b9a8541dd712b1fae043d13dea8674/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/554f0b9b-bae8-478b-b3b1-85b376a79cc9/volumes/kubernetes.io~projected/kube-api-access-7jtv5 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a948657-c616-435c-90db-b880a0db5bd9/volumes/kubernetes.io~projected/kube-api-access-lmctv DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/46289f03-406c-4831-9ad1-0e6751fd7bd5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d23976a-d44f-4b71-8b58-04932007cc17/volumes/kubernetes.io~projected/kube-api-access-5p7nr DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/778f7f74-6134-47ab-a978-cf3462316e72/volumes/kubernetes.io~projected/kube-api-access-xpwfh DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/363a0adc-193f-48d3-9628-1b9d10665597/volumes/kubernetes.io~projected/kube-api-access-2jzbb DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f42e28b337da0af37c779d0fbb1fd17de5d3be8af05390d241dd9e63739784f/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e/volumes/kubernetes.io~projected/kube-api-access-l95t8 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ecca7198a975105584690498b9ce1123acee1a0c6317cc908f4f27982599bcc2/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/260d2659-42a8-461f-8994-0c5f9522a80d/volumes/kubernetes.io~projected/kube-api-access-h8ccv DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b2937779-b03d-4a5b-87bc-f062283eeb2e/volumes/kubernetes.io~projected/kube-api-access-bx8v8 DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/54b4e7b90dcf8224029f72fe0f07d265dbfcf68e7392a34072a420cc3cb09f76/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4028eaff961969401c3554706f9bf3f13c181ffa0560bcbc20040c0aef366984/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e433332d61097a077363fdf206e7b57cbabb4a1b00b56772df19d032b73be9e3/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/778f7f74-6134-47ab-a978-cf3462316e72/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/617427c7-590d-4d80-8c4e-e11c7812dca0/volumes/kubernetes.io~projected/kube-api-access-5jhg2 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e/volumes/kubernetes.io~projected/kube-api-access-hj598 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd1ee92b2bc84070ab1e222d12ad2419f4a480741ac772a445913c7eb1e2543c/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b6e449a33b18a7662a942a7cf450cb9b06ddb9743afb5dc310f3422a7cd2dab/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:3545637cdbb8177 MacAddress:d6:01:40:30:89:a0 Speed:10000 Mtu:8901} {Name:3f42e28b337da0a MacAddress:06:b4:82:f6:da:b5 Speed:10000 Mtu:8901} {Name:8924ad7e7609d36 MacAddress:a2:d7:8d:52:a7:52 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d0:e5:f2:c9 Speed:0 Mtu:9001} {Name:br-int MacAddress:d6:43:ce:85:8c:04 Speed:0 Mtu:8901} {Name:daca8037d7968a5 MacAddress:fa:c3:57:f6:f1:74 Speed:10000 Mtu:8901} {Name:ecca7198a975105 MacAddress:82:b6:ab:b4:66:90 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d0:e5:f2:c9 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:1a:ec:f6:b4:8a:26 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:66:d9:ea:e9:f1:7d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.032662 4468 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.032708 4468 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.032993 4468 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.033010 4468 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-76.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.033160 4468 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.033169 4468 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.033179 4468 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.033198 4468 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.033268 4468 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.033440 4468 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.033544 4468 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.033582 4468 kubelet.go:493] "Attempting to sync node with API server" Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.033593 4468 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.033603 4468 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.033611 4468 kubelet.go:405] "Adding apiserver pod source" Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.033620 4468 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.034382 4468 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.034396 4468 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 11:54:09.035334 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.034918 4468 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 11:54:09.038047 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.037783 4468 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 11:54:09.038047 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.037804 4468 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 11:54:09.038047 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.037938 4468 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 11:54:09.038345 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.038322 4468 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 11:54:09.038415 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.038386 4468 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 11:54:09.038522 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.038509 4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 11:54:09.038571 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.038529 4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 11:54:09.038571 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.038540 4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 11:54:09.038571 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.038557 4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 11:54:09.038571 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.038566 4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 11:54:09.038730 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.038575 4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 11:54:09.038730 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.038586 4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 11:54:09.038730 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.038596 4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 11:54:09.038730 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.038608 4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 11:54:09.038730 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.038617 4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 11:54:09.038730 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.038630 4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 11:54:09.038730 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.038644 4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 11:54:09.038730 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.038676 4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 11:54:09.038730 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.038686 4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 11:54:09.043129 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.043113 4468 server.go:1263] "Started kubelet" Oct 30 11:54:09.043595 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.043210 4468 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 11:54:09.043595 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.043393 4468 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 11:54:09.043595 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.043464 4468 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 11:54:09.043640 ip-10-0-131-76 systemd[1]: Started Kubernetes Kubelet. Oct 30 11:54:09.044233 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.044154 4468 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 11:54:09.044647 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.044631 4468 server.go:317] "Adding debug handlers to kubelet server" Oct 30 11:54:09.045748 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.045731 4468 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 11:54:09.045943 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.045847 4468 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 11:54:09.047667 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.046515 4468 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 11:48:31 +0000 UTC" deadline="2028-08-09 08:57:32.803572524 +0000 UTC" Oct 30 11:54:09.047667 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.046556 4468 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15573h3m23.757019693s" Oct 30 11:54:09.047667 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:09.046588 4468 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-76.ec2.internal\" not found" Oct 30 11:54:09.047667 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.047240 4468 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 11:54:09.047667 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.047439 4468 factory.go:55] Registering systemd factory Oct 30 11:54:09.047667 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.047452 4468 factory.go:223] Registration of the systemd container factory successfully Oct 30 11:54:09.048686 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.048082 4468 factory.go:153] Registering CRI-O factory Oct 30 11:54:09.048686 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.048096 4468 factory.go:223] Registration of the crio container factory successfully Oct 30 11:54:09.048686 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.048148 4468 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 11:54:09.048686 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.048166 4468 factory.go:103] Registering Raw factory Oct 30 11:54:09.048686 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.048180 4468 manager.go:1199] Started watching for new ooms in manager Oct 30 11:54:09.048686 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.048508 4468 manager.go:319] Starting recovery of all containers Oct 30 11:54:09.050245 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.049944 4468 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 11:54:09.050245 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.049980 4468 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051753 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="778f7f74-6134-47ab-a978-cf3462316e72" volumeName="kubernetes.io/projected/778f7f74-6134-47ab-a978-cf3462316e72-kube-api-access-xpwfh" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051782 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d897418e-880a-46a9-8b13-9c2887bfddb7" volumeName="kubernetes.io/configmap/d897418e-880a-46a9-8b13-9c2887bfddb7-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051792 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a948657-c616-435c-90db-b880a0db5bd9" volumeName="kubernetes.io/projected/1a948657-c616-435c-90db-b880a0db5bd9-kube-api-access-lmctv" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051799 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="260d2659-42a8-461f-8994-0c5f9522a80d" volumeName="kubernetes.io/projected/260d2659-42a8-461f-8994-0c5f9522a80d-kube-api-access-h8ccv" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051808 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f2f3c38-f710-491a-9728-7c641291d679" volumeName="kubernetes.io/configmap/4f2f3c38-f710-491a-9728-7c641291d679-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051815 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf0f7768-5ba5-4b50-a601-0c4202b2b66e" volumeName="kubernetes.io/projected/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-kube-api-access-6lswd" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051822 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d897418e-880a-46a9-8b13-9c2887bfddb7" volumeName="kubernetes.io/configmap/d897418e-880a-46a9-8b13-9c2887bfddb7-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051830 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a948657-c616-435c-90db-b880a0db5bd9" volumeName="kubernetes.io/configmap/1a948657-c616-435c-90db-b880a0db5bd9-config-volume" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051849 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="554f0b9b-bae8-478b-b3b1-85b376a79cc9" volumeName="kubernetes.io/configmap/554f0b9b-bae8-478b-b3b1-85b376a79cc9-env-overrides" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051857 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82a77175-bb28-474f-b4ec-dac73d790410" volumeName="kubernetes.io/projected/82a77175-bb28-474f-b4ec-dac73d790410-kube-api-access-sp7n7" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051865 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d23976a-d44f-4b71-8b58-04932007cc17" volumeName="kubernetes.io/secret/9d23976a-d44f-4b71-8b58-04932007cc17-cert" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051872 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f15ba445-e38a-46f6-be0f-65f7307e79d6" volumeName="kubernetes.io/secret/f15ba445-e38a-46f6-be0f-65f7307e79d6-agent-certs" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051888 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e" volumeName="kubernetes.io/configmap/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e-serviceca" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051896 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31df51a8-4f22-4745-b25f-8b69654f2845" volumeName="kubernetes.io/projected/31df51a8-4f22-4745-b25f-8b69654f2845-kube-api-access-v8tg9" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051905 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46289f03-406c-4831-9ad1-0e6751fd7bd5" volumeName="kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051912 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d23976a-d44f-4b71-8b58-04932007cc17" volumeName="kubernetes.io/projected/9d23976a-d44f-4b71-8b58-04932007cc17-kube-api-access-5p7nr" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051920 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf0f7768-5ba5-4b50-a601-0c4202b2b66e" volumeName="kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-trusted-ca-bundle" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051927 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d897418e-880a-46a9-8b13-9c2887bfddb7" volumeName="kubernetes.io/projected/d897418e-880a-46a9-8b13-9c2887bfddb7-kube-api-access-s5g55" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051934 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a948657-c616-435c-90db-b880a0db5bd9" volumeName="kubernetes.io/empty-dir/1a948657-c616-435c-90db-b880a0db5bd9-tmp-dir" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051941 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="363a0adc-193f-48d3-9628-1b9d10665597" volumeName="kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051948 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="554f0b9b-bae8-478b-b3b1-85b376a79cc9" volumeName="kubernetes.io/configmap/554f0b9b-bae8-478b-b3b1-85b376a79cc9-ovnkube-config" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051956 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2937779-b03d-4a5b-87bc-f062283eeb2e" volumeName="kubernetes.io/configmap/b2937779-b03d-4a5b-87bc-f062283eeb2e-iptables-alerter-script" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051963 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf0f7768-5ba5-4b50-a601-0c4202b2b66e" volumeName="kubernetes.io/empty-dir/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-archives-path" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051970 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f15ba445-e38a-46f6-be0f-65f7307e79d6" volumeName="kubernetes.io/configmap/f15ba445-e38a-46f6-be0f-65f7307e79d6-konnectivity-ca" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051976 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31df51a8-4f22-4745-b25f-8b69654f2845" volumeName="kubernetes.io/configmap/31df51a8-4f22-4745-b25f-8b69654f2845-multus-daemon-config" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051983 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f2f3c38-f710-491a-9728-7c641291d679" volumeName="kubernetes.io/empty-dir/4f2f3c38-f710-491a-9728-7c641291d679-data-volume" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051991 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="554f0b9b-bae8-478b-b3b1-85b376a79cc9" volumeName="kubernetes.io/secret/554f0b9b-bae8-478b-b3b1-85b376a79cc9-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.051997 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="778f7f74-6134-47ab-a978-cf3462316e72" volumeName="kubernetes.io/configmap/778f7f74-6134-47ab-a978-cf3462316e72-signing-cabundle" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.052013 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="778f7f74-6134-47ab-a978-cf3462316e72" volumeName="kubernetes.io/secret/778f7f74-6134-47ab-a978-cf3462316e72-signing-key" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.052020 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11f0fad5-3b17-44fe-a2e0-4e99f496ab6e" volumeName="kubernetes.io/empty-dir/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-tmp" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.052030 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11f0fad5-3b17-44fe-a2e0-4e99f496ab6e" volumeName="kubernetes.io/projected/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-kube-api-access-hj598" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.052038 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="554f0b9b-bae8-478b-b3b1-85b376a79cc9" volumeName="kubernetes.io/configmap/554f0b9b-bae8-478b-b3b1-85b376a79cc9-ovnkube-script-lib" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.052045 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="617427c7-590d-4d80-8c4e-e11c7812dca0" volumeName="kubernetes.io/empty-dir/617427c7-590d-4d80-8c4e-e11c7812dca0-tmp-dir" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.052051 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="617427c7-590d-4d80-8c4e-e11c7812dca0" volumeName="kubernetes.io/projected/617427c7-590d-4d80-8c4e-e11c7812dca0-kube-api-access-5jhg2" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.052059 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2937779-b03d-4a5b-87bc-f062283eeb2e" volumeName="kubernetes.io/projected/b2937779-b03d-4a5b-87bc-f062283eeb2e-kube-api-access-bx8v8" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.052086 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11f0fad5-3b17-44fe-a2e0-4e99f496ab6e" volumeName="kubernetes.io/empty-dir/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-tuned" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.052098 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f2f3c38-f710-491a-9728-7c641291d679" volumeName="kubernetes.io/projected/4f2f3c38-f710-491a-9728-7c641291d679-kube-api-access-nxpq6" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.052106 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="554f0b9b-bae8-478b-b3b1-85b376a79cc9" volumeName="kubernetes.io/projected/554f0b9b-bae8-478b-b3b1-85b376a79cc9-kube-api-access-7jtv5" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.052113 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf0f7768-5ba5-4b50-a601-0c4202b2b66e" volumeName="kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-service-ca-bundle" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.052119 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d897418e-880a-46a9-8b13-9c2887bfddb7" volumeName="kubernetes.io/configmap/d897418e-880a-46a9-8b13-9c2887bfddb7-cni-binary-copy" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.052127 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e" volumeName="kubernetes.io/projected/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e-kube-api-access-l95t8" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.052134 4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31df51a8-4f22-4745-b25f-8b69654f2845" volumeName="kubernetes.io/configmap/31df51a8-4f22-4745-b25f-8b69654f2845-cni-binary-copy" seLinuxMountContext="" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.052142 4468 reconstruct.go:97] "Volume reconstruction finished" Oct 30 11:54:09.053934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.052147 4468 reconciler.go:29] "Reconciler: start to sync state" Oct 30 11:54:09.079463 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.079428 4468 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 11:54:09.081420 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.081398 4468 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 11:54:09.081420 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.081422 4468 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 11:54:09.081599 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.081442 4468 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 11:54:09.081599 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:09.081476 4468 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 11:54:09.099336 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.099303 4468 generic.go:356] "Generic (PLEG): container finished" podID="d897418e-880a-46a9-8b13-9c2887bfddb7" containerID="e3d17439b9adf04c3d12a7140cf0a85ef10f0718789d076e25c554df31ded05e" exitCode=0 Oct 30 11:54:09.099336 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.099330 4468 generic.go:356] "Generic (PLEG): container finished" podID="d897418e-880a-46a9-8b13-9c2887bfddb7" containerID="40ca657fc904a45b0a291858c0054f6b5194b92c21a0309f77e9f6e7233b454a" exitCode=0 Oct 30 11:54:09.099336 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.099339 4468 generic.go:356] "Generic (PLEG): container finished" podID="d897418e-880a-46a9-8b13-9c2887bfddb7" containerID="2037fa397d756b817c26e761a4f174416df9e18f905d2f93316da1cb9a595cf7" exitCode=0 Oct 30 11:54:09.099336 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.099344 4468 generic.go:356] "Generic (PLEG): container finished" podID="d897418e-880a-46a9-8b13-9c2887bfddb7" containerID="06375939516acac61587d3a01ee9a13dbf948dae058996d53432921f44a71f34" exitCode=0 Oct 30 11:54:09.099628 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.099350 4468 generic.go:356] "Generic (PLEG): container finished" podID="d897418e-880a-46a9-8b13-9c2887bfddb7" containerID="36d3aec6dee2041d7691fa089efced35c8034a425e6ab7cd14af6d6afce36fa3" exitCode=0 Oct 30 11:54:09.099628 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.099356 4468 generic.go:356] "Generic (PLEG): container finished" podID="d897418e-880a-46a9-8b13-9c2887bfddb7" containerID="5e6e6fd4e22259a168f79fa076929dda8721e979ddfebb85d211563113410389" exitCode=0 Oct 30 11:54:09.101746 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.101716 4468 generic.go:356] "Generic (PLEG): container finished" podID="ac067dd06bb744f9ffd35d4622ecd99a" containerID="14e8486f873c3bf5d098b8a0d13f2e16042e85a72e0af040ce7ba8dbe87636c7" exitCode=0 Oct 30 11:54:09.102162 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.102146 4468 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 11:54:09.103879 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.103854 4468 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:54:09.103879 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.103868 4468 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:54:09.104273 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.104254 4468 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:54:09.115701 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.115669 4468 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:54:09.120611 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.120582 4468 manager.go:324] Recovery completed Oct 30 11:54:09.137103 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.137017 4468 cpu_manager.go:226] "Starting" policy="none" Oct 30 11:54:09.137103 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.137034 4468 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 11:54:09.137103 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.137052 4468 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 11:54:09.137289 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.137221 4468 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 11:54:09.137289 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.137233 4468 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 11:54:09.137289 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.137258 4468 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 11:54:09.137289 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.137263 4468 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 11:54:09.137289 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.137268 4468 policy_none.go:50] "Start" Oct 30 11:54:09.137289 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.137277 4468 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 11:54:09.137289 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.137286 4468 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 11:54:09.137470 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.137417 4468 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 11:54:09.137470 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.137424 4468 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 11:54:09.137470 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.137430 4468 policy_none.go:44] "Start" Oct 30 11:54:09.141553 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.141516 4468 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 11:54:09.146376 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.146359 4468 manager.go:341] "Starting Device Plugin manager" Oct 30 11:54:09.146916 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:09.146390 4468 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 11:54:09.146986 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.146918 4468 server.go:87] "Starting device plugin registration server" Oct 30 11:54:09.147187 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.147176 4468 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 11:54:09.147244 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.147191 4468 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 11:54:09.147301 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.147285 4468 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 11:54:09.147405 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.147378 4468 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 11:54:09.147446 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.147413 4468 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 11:54:09.147446 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.147420 4468 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 11:54:09.147555 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.147470 4468 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T11:54:09.147402202Z","UUID":"c80c6e50-7153-45c4-8401-d4a6c74611ff","Handler":null,"Name":"","Endpoint":""} Oct 30 11:54:09.148667 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:09.148645 4468 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 11:54:09.148844 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.148829 4468 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 11:54:09.148919 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.148856 4468 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 11:54:09.181712 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.181659 4468 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal"] Oct 30 11:54:09.182266 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.182202 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" event={"ID":"ac067dd06bb744f9ffd35d4622ecd99a","Type":"ContainerStarted","Data":"f911c175e62302e81ef9da5ce4089af7daf6745fc776f535f6b8902071ca0e01"} Oct 30 11:54:09.182386 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.182272 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" event={"ID":"ac067dd06bb744f9ffd35d4622ecd99a","Type":"ContainerDied","Data":"14e8486f873c3bf5d098b8a0d13f2e16042e85a72e0af040ce7ba8dbe87636c7"} Oct 30 11:54:09.182386 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.182292 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" event={"ID":"ac067dd06bb744f9ffd35d4622ecd99a","Type":"ContainerStarted","Data":"4028eaff961969401c3554706f9bf3f13c181ffa0560bcbc20040c0aef366984"} Oct 30 11:54:09.182386 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.182309 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal" event={"ID":"5a981c242c496637c2b85dbce301f463","Type":"ContainerStarted","Data":"45de1b8a7e7af25821a185f467e0166bb199a69f0be0feca8643b3166e804e90"} Oct 30 11:54:09.182386 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.182314 4468 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" Oct 30 11:54:09.182386 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.182328 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal" event={"ID":"5a981c242c496637c2b85dbce301f463","Type":"ContainerStarted","Data":"eee0a121e54d3c931a8ffcd5feed923fac046341f5f69d4af853f871c712bd1a"} Oct 30 11:54:09.182623 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.182418 4468 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal" Oct 30 11:54:09.208631 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.208601 4468 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:54:09.208749 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:09.208654 4468 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-76.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal" Oct 30 11:54:09.209296 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.209284 4468 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:54:09.209350 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:09.209310 4468 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" Oct 30 11:54:09.248193 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.248155 4468 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:54:09.249557 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.249536 4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:54:09.249660 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.249573 4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:54:09.249660 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.249587 4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-76.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:54:09.249660 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.249634 4468 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-76.ec2.internal" Oct 30 11:54:09.286795 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.286768 4468 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-76.ec2.internal" Oct 30 11:54:09.286929 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.286846 4468 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-76.ec2.internal" Oct 30 11:54:09.303378 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.303352 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac067dd06bb744f9ffd35d4622ecd99a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal\" (UID: \"ac067dd06bb744f9ffd35d4622ecd99a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" Oct 30 11:54:09.303511 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.303386 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a981c242c496637c2b85dbce301f463-config\") pod \"kube-apiserver-proxy-ip-10-0-131-76.ec2.internal\" (UID: \"5a981c242c496637c2b85dbce301f463\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal" Oct 30 11:54:09.303511 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.303417 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac067dd06bb744f9ffd35d4622ecd99a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal\" (UID: \"ac067dd06bb744f9ffd35d4622ecd99a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" Oct 30 11:54:09.404624 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.404542 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac067dd06bb744f9ffd35d4622ecd99a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal\" (UID: \"ac067dd06bb744f9ffd35d4622ecd99a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" Oct 30 11:54:09.404624 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.404583 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac067dd06bb744f9ffd35d4622ecd99a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal\" (UID: \"ac067dd06bb744f9ffd35d4622ecd99a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" Oct 30 11:54:09.404624 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.404600 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a981c242c496637c2b85dbce301f463-config\") pod \"kube-apiserver-proxy-ip-10-0-131-76.ec2.internal\" (UID: \"5a981c242c496637c2b85dbce301f463\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal" Oct 30 11:54:09.404824 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.404652 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a981c242c496637c2b85dbce301f463-config\") pod \"kube-apiserver-proxy-ip-10-0-131-76.ec2.internal\" (UID: \"5a981c242c496637c2b85dbce301f463\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal" Oct 30 11:54:09.404824 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.404657 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac067dd06bb744f9ffd35d4622ecd99a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal\" (UID: \"ac067dd06bb744f9ffd35d4622ecd99a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" Oct 30 11:54:09.404824 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:09.404700 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac067dd06bb744f9ffd35d4622ecd99a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal\" (UID: \"ac067dd06bb744f9ffd35d4622ecd99a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" Oct 30 11:54:10.034325 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.034291 4468 apiserver.go:52] "Watching apiserver" Oct 30 11:54:10.080614 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.080583 4468 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 11:54:10.081108 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.081057 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-qq48w","openshift-dns/dns-default-p4nvb","openshift-cluster-node-tuning-operator/tuned-2jgjj","openshift-insights/insights-runtime-extractor-6dvql","openshift-service-ca/service-ca-5fffc97bdf-gqgxk","kube-system/konnectivity-agent-qjvts","openshift-dns/node-resolver-nhbdl","openshift-insights/periodic-gathering-nr74p-m5v7z","openshift-multus/multus-6pxtt","kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal","openshift-image-registry/node-ca-7c297","openshift-multus/multus-additional-cni-plugins-d7bvp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6","kube-system/global-pull-secret-syncer-mqv9x","openshift-ingress-canary/ingress-canary-z6hh6","openshift-multus/network-metrics-daemon-rh7rw","openshift-network-diagnostics/network-check-target-ld8pr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal","openshift-network-operator/iptables-alerter-htqhw"] Oct 30 11:54:10.083517 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.083497 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:54:10.087552 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.087533 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:10.087903 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.087890 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p4nvb" Oct 30 11:54:10.104301 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.104270 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 11:54:10.104301 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.104279 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6hs8x\"" Oct 30 11:54:10.104690 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.104672 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pxw6w\"" Oct 30 11:54:10.105164 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.105151 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sw2hc\"" Oct 30 11:54:10.107683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.107667 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tz47t\"" Oct 30 11:54:10.108868 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.108854 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 11:54:10.109168 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.109152 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 11:54:10.110956 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.110936 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 11:54:10.111031 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.110977 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 11:54:10.111031 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.110984 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 11:54:10.111031 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.111023 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 11:54:10.111177 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.111110 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:54:10.111177 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.111132 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 11:54:10.114050 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.114038 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:54:10.114106 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.114085 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 11:54:10.114149 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.114137 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 11:54:10.114585 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.114570 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 11:54:10.114638 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.114589 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 11:54:10.118579 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.118566 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 11:54:10.118633 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.118619 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 11:54:10.118869 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.118855 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 11:54:10.119561 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.119548 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 11:54:10.120077 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.120054 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 11:54:10.120201 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.120187 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 11:54:10.120253 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.120221 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lrxbh\"" Oct 30 11:54:10.120301 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.120256 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 11:54:10.120301 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.120263 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 11:54:10.120301 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.120295 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 11:54:10.120556 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.120543 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 11:54:10.120647 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.120635 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 11:54:10.120939 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.120928 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 11:54:10.121059 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.121043 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 11:54:10.121228 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.121120 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 11:54:10.121337 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.121323 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 11:54:10.121380 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.121329 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 11:54:10.121380 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.121335 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 11:54:10.121459 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.121446 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rpjw7\"" Oct 30 11:54:10.121511 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.121479 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 11:54:10.121690 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.121679 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 11:54:10.121767 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.121755 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 11:54:10.124532 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.124516 4468 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" Oct 30 11:54:10.124635 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.124621 4468 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal" Oct 30 11:54:10.130093 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.130063 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 11:54:10.130167 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.130153 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tdxpc\"" Oct 30 11:54:10.131669 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.131631 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 11:54:10.131748 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.131736 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7cm7r\"" Oct 30 11:54:10.132238 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.132220 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 11:54:10.132238 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.132234 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 11:54:10.132330 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.132246 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 11:54:10.132330 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.132281 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-847hl\"" Oct 30 11:54:10.132330 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.132298 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 11:54:10.133008 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.132996 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-96ksc\"" Oct 30 11:54:10.133680 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.133665 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 11:54:10.133722 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.133701 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 11:54:10.133752 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.133723 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjl8c\"" Oct 30 11:54:10.134567 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.134555 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 11:54:10.134615 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.134579 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 11:54:10.134615 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.134595 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 11:54:10.134615 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.134604 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zfblq\"" Oct 30 11:54:10.138287 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.138274 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8n4dr\"" Oct 30 11:54:10.138361 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.138278 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8pg5x\"" Oct 30 11:54:10.138361 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.138322 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jjcjh\"" Oct 30 11:54:10.138361 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.138338 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 11:54:10.138361 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.138348 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jxkrq\"" Oct 30 11:54:10.138491 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.138375 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-llkx2\"" Oct 30 11:54:10.147479 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.147462 4468 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:54:10.147579 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:10.147507 4468 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-76.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-76.ec2.internal" Oct 30 11:54:10.148350 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.148336 4468 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:54:10.148406 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:10.148375 4468 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-76.ec2.internal" Oct 30 11:54:10.148568 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.148556 4468 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 11:54:10.208017 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.207982 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-etc-selinux\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.208017 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208013 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-cnibin\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.208209 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208044 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-hostroot\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.208209 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208127 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e-host\") pod \"node-ca-7c297\" (UID: \"29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e\") " pod="openshift-image-registry/node-ca-7c297" Oct 30 11:54:10.208209 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208160 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d897418e-880a-46a9-8b13-9c2887bfddb7-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:54:10.208209 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208187 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46289f03-406c-4831-9ad1-0e6751fd7bd5-dbus\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:54:10.208346 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208217 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/554f0b9b-bae8-478b-b3b1-85b376a79cc9-ovn-node-metrics-cert\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.208346 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208276 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-sysconfig\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.208346 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208301 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-run\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.208346 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208325 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5g55\" (UniqueName: \"kubernetes.io/projected/d897418e-880a-46a9-8b13-9c2887bfddb7-kube-api-access-s5g55\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:54:10.208346 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208346 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8ccv\" (UniqueName: \"kubernetes.io/projected/260d2659-42a8-461f-8994-0c5f9522a80d-kube-api-access-h8ccv\") pod \"network-metrics-daemon-rh7rw\" (UID: \"260d2659-42a8-461f-8994-0c5f9522a80d\") " pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:54:10.208597 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208379 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-run-openvswitch\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.208597 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208410 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f2f3c38-f710-491a-9728-7c641291d679-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:10.208597 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208438 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lswd\" (UniqueName: \"kubernetes.io/projected/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-kube-api-access-6lswd\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:54:10.208597 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208444 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/554f0b9b-bae8-478b-b3b1-85b376a79cc9-ovn-node-metrics-cert\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.208597 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208444 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d897418e-880a-46a9-8b13-9c2887bfddb7-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:54:10.208597 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208467 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/617427c7-590d-4d80-8c4e-e11c7812dca0-tmp-dir\") pod \"node-resolver-nhbdl\" (UID: \"617427c7-590d-4d80-8c4e-e11c7812dca0\") " pod="openshift-dns/node-resolver-nhbdl" Oct 30 11:54:10.208597 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208494 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-run-netns\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.208809 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208642 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8tg9\" (UniqueName: \"kubernetes.io/projected/31df51a8-4f22-4745-b25f-8b69654f2845-kube-api-access-v8tg9\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.208809 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208664 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/617427c7-590d-4d80-8c4e-e11c7812dca0-tmp-dir\") pod \"node-resolver-nhbdl\" (UID: \"617427c7-590d-4d80-8c4e-e11c7812dca0\") " pod="openshift-dns/node-resolver-nhbdl" Oct 30 11:54:10.208809 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208674 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46289f03-406c-4831-9ad1-0e6751fd7bd5-kubelet-config\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:54:10.208809 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208681 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f2f3c38-f710-491a-9728-7c641291d679-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:10.208809 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208718 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-run-systemd\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.208809 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208776 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-sysctl-conf\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.208809 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208804 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-host\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.209085 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.208850 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpwfh\" (UniqueName: \"kubernetes.io/projected/778f7f74-6134-47ab-a978-cf3462316e72-kube-api-access-xpwfh\") pod \"service-ca-5fffc97bdf-gqgxk\" (UID: \"778f7f74-6134-47ab-a978-cf3462316e72\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" Oct 30 11:54:10.209085 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209039 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d897418e-880a-46a9-8b13-9c2887bfddb7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:54:10.209186 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209086 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/778f7f74-6134-47ab-a978-cf3462316e72-signing-key\") pod \"service-ca-5fffc97bdf-gqgxk\" (UID: \"778f7f74-6134-47ab-a978-cf3462316e72\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" Oct 30 11:54:10.209186 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209117 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-run-multus-certs\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.209186 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209147 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d897418e-880a-46a9-8b13-9c2887bfddb7-system-cni-dir\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:54:10.209186 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209175 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.209370 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209254 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d897418e-880a-46a9-8b13-9c2887bfddb7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:54:10.209370 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209322 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpq6\" (UniqueName: \"kubernetes.io/projected/4f2f3c38-f710-491a-9728-7c641291d679-kube-api-access-nxpq6\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:10.209464 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209366 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-var-lib-openvswitch\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.209464 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209377 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/778f7f74-6134-47ab-a978-cf3462316e72-signing-key\") pod \"service-ca-5fffc97bdf-gqgxk\" (UID: \"778f7f74-6134-47ab-a978-cf3462316e72\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" Oct 30 11:54:10.209464 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209387 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-cni-bin\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.209464 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209421 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-sys\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.209464 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209450 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-tuned\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.209693 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209480 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jhg2\" (UniqueName: \"kubernetes.io/projected/617427c7-590d-4d80-8c4e-e11c7812dca0-kube-api-access-5jhg2\") pod \"node-resolver-nhbdl\" (UID: \"617427c7-590d-4d80-8c4e-e11c7812dca0\") " pod="openshift-dns/node-resolver-nhbdl" Oct 30 11:54:10.209693 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209549 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-var-lib-cni-multus\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.209693 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209564 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-multus-conf-dir\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.209693 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209567 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-tuned\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.209693 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209583 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:54:10.209693 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209645 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-run-ovn\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.209693 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209668 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-archives-path\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:54:10.209693 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209684 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-var-lib-kubelet\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.210043 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209700 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jzbb\" (UniqueName: \"kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb\") pod \"network-check-target-ld8pr\" (UID: \"363a0adc-193f-48d3-9628-1b9d10665597\") " pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:54:10.210043 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209715 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-system-cni-dir\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.210043 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209735 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx8v8\" (UniqueName: \"kubernetes.io/projected/b2937779-b03d-4a5b-87bc-f062283eeb2e-kube-api-access-bx8v8\") pod \"iptables-alerter-htqhw\" (UID: \"b2937779-b03d-4a5b-87bc-f062283eeb2e\") " pod="openshift-network-operator/iptables-alerter-htqhw" Oct 30 11:54:10.210043 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209756 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-archives-path\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:54:10.210043 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209793 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-etc-openvswitch\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.210043 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209838 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-log-socket\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.210043 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209853 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46289f03-406c-4831-9ad1-0e6751fd7bd5-original-pull-secret\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:54:10.210043 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209863 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-run-ovn-kubernetes\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.210043 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209900 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-tmp\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.210043 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209920 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/778f7f74-6134-47ab-a978-cf3462316e72-signing-cabundle\") pod \"service-ca-5fffc97bdf-gqgxk\" (UID: \"778f7f74-6134-47ab-a978-cf3462316e72\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" Oct 30 11:54:10.210043 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209964 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/617427c7-590d-4d80-8c4e-e11c7812dca0-hosts-file\") pod \"node-resolver-nhbdl\" (UID: \"617427c7-590d-4d80-8c4e-e11c7812dca0\") " pod="openshift-dns/node-resolver-nhbdl" Oct 30 11:54:10.210043 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.209984 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-tmp\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.210043 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210014 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d897418e-880a-46a9-8b13-9c2887bfddb7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210058 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b2937779-b03d-4a5b-87bc-f062283eeb2e-iptables-alerter-script\") pod \"iptables-alerter-htqhw\" (UID: \"b2937779-b03d-4a5b-87bc-f062283eeb2e\") " pod="openshift-network-operator/iptables-alerter-htqhw" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210097 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/778f7f74-6134-47ab-a978-cf3462316e72-signing-cabundle\") pod \"service-ca-5fffc97bdf-gqgxk\" (UID: \"778f7f74-6134-47ab-a978-cf3462316e72\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210116 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs\") pod \"network-metrics-daemon-rh7rw\" (UID: \"260d2659-42a8-461f-8994-0c5f9522a80d\") " pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210161 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-node-log\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210188 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-kubernetes\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210203 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-run-k8s-cni-cncf-io\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210236 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210271 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d897418e-880a-46a9-8b13-9c2887bfddb7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210294 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2937779-b03d-4a5b-87bc-f062283eeb2e-host-slash\") pod \"iptables-alerter-htqhw\" (UID: \"b2937779-b03d-4a5b-87bc-f062283eeb2e\") " pod="openshift-network-operator/iptables-alerter-htqhw" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210317 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-registration-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210333 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-device-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210343 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b2937779-b03d-4a5b-87bc-f062283eeb2e-iptables-alerter-script\") pod \"iptables-alerter-htqhw\" (UID: \"b2937779-b03d-4a5b-87bc-f062283eeb2e\") " pod="openshift-network-operator/iptables-alerter-htqhw" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210365 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31df51a8-4f22-4745-b25f-8b69654f2845-cni-binary-copy\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210394 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-var-lib-cni-bin\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210411 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a948657-c616-435c-90db-b880a0db5bd9-tmp-dir\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210429 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d897418e-880a-46a9-8b13-9c2887bfddb7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210453 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e-serviceca\") pod \"node-ca-7c297\" (UID: \"29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e\") " pod="openshift-image-registry/node-ca-7c297" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210491 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a948657-c616-435c-90db-b880a0db5bd9-tmp-dir\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210507 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l95t8\" (UniqueName: \"kubernetes.io/projected/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e-kube-api-access-l95t8\") pod \"node-ca-7c297\" (UID: \"29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e\") " pod="openshift-image-registry/node-ca-7c297" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210526 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-systemd-units\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210542 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/554f0b9b-bae8-478b-b3b1-85b376a79cc9-ovnkube-script-lib\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210557 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f2f3c38-f710-491a-9728-7c641291d679-crio-socket\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210572 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210579 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31df51a8-4f22-4745-b25f-8b69654f2845-cni-binary-copy\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210605 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e-serviceca\") pod \"node-ca-7c297\" (UID: \"29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e\") " pod="openshift-image-registry/node-ca-7c297" Oct 30 11:54:10.210683 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210610 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-service-ca-bundle\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:54:10.211766 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.210677 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-sysctl-d\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.211766 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.211031 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f15ba445-e38a-46f6-be0f-65f7307e79d6-konnectivity-ca\") pod \"konnectivity-agent-qjvts\" (UID: \"f15ba445-e38a-46f6-be0f-65f7307e79d6\") " pod="kube-system/konnectivity-agent-qjvts" Oct 30 11:54:10.211766 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.211085 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-run-netns\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.211766 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.211139 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/554f0b9b-bae8-478b-b3b1-85b376a79cc9-ovnkube-script-lib\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.211766 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.211457 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/554f0b9b-bae8-478b-b3b1-85b376a79cc9-env-overrides\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.211766 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.211512 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f15ba445-e38a-46f6-be0f-65f7307e79d6-konnectivity-ca\") pod \"konnectivity-agent-qjvts\" (UID: \"f15ba445-e38a-46f6-be0f-65f7307e79d6\") " pod="kube-system/konnectivity-agent-qjvts" Oct 30 11:54:10.212020 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.211827 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-service-ca-bundle\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:54:10.213907 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.213881 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/554f0b9b-bae8-478b-b3b1-85b376a79cc9-env-overrides\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.214036 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.213921 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-systemd\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.214036 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.213953 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp7n7\" (UniqueName: \"kubernetes.io/projected/82a77175-bb28-474f-b4ec-dac73d790410-kube-api-access-sp7n7\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.214036 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.214006 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/31df51a8-4f22-4745-b25f-8b69654f2845-multus-daemon-config\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.214221 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.214093 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-etc-kubernetes\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.214221 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.214142 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmctv\" (UniqueName: \"kubernetes.io/projected/1a948657-c616-435c-90db-b880a0db5bd9-kube-api-access-lmctv\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:54:10.214221 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.214177 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-cni-netd\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.214221 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.214208 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jtv5\" (UniqueName: \"kubernetes.io/projected/554f0b9b-bae8-478b-b3b1-85b376a79cc9-kube-api-access-7jtv5\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.214411 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.214242 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-modprobe-d\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.214411 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.214275 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj598\" (UniqueName: \"kubernetes.io/projected/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-kube-api-access-hj598\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.214411 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.214386 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/31df51a8-4f22-4745-b25f-8b69654f2845-multus-daemon-config\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.214767 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.214742 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a948657-c616-435c-90db-b880a0db5bd9-config-volume\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:54:10.214834 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.214788 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f15ba445-e38a-46f6-be0f-65f7307e79d6-agent-certs\") pod \"konnectivity-agent-qjvts\" (UID: \"f15ba445-e38a-46f6-be0f-65f7307e79d6\") " pod="kube-system/konnectivity-agent-qjvts" Oct 30 11:54:10.214834 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.214816 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d897418e-880a-46a9-8b13-9c2887bfddb7-cnibin\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:54:10.214941 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.214842 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/554f0b9b-bae8-478b-b3b1-85b376a79cc9-ovnkube-config\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.214992 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.214938 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-trusted-ca-bundle\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:54:10.214992 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.214975 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-lib-modules\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.215108 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.215000 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-multus-cni-dir\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.215108 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.215029 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.215108 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.215032 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a948657-c616-435c-90db-b880a0db5bd9-config-volume\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:54:10.215108 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.215054 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-sys-fs\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.215108 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.215063 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f15ba445-e38a-46f6-be0f-65f7307e79d6-agent-certs\") pod \"konnectivity-agent-qjvts\" (UID: \"f15ba445-e38a-46f6-be0f-65f7307e79d6\") " pod="kube-system/konnectivity-agent-qjvts" Oct 30 11:54:10.215108 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.215096 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/554f0b9b-bae8-478b-b3b1-85b376a79cc9-ovnkube-config\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.215380 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.215135 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-multus-socket-dir-parent\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.215380 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.215201 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-var-lib-kubelet\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.215380 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.215240 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d23976a-d44f-4b71-8b58-04932007cc17-cert\") pod \"ingress-canary-z6hh6\" (UID: \"9d23976a-d44f-4b71-8b58-04932007cc17\") " pod="openshift-ingress-canary/ingress-canary-z6hh6" Oct 30 11:54:10.215380 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.215268 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p7nr\" (UniqueName: \"kubernetes.io/projected/9d23976a-d44f-4b71-8b58-04932007cc17-kube-api-access-5p7nr\") pod \"ingress-canary-z6hh6\" (UID: \"9d23976a-d44f-4b71-8b58-04932007cc17\") " pod="openshift-ingress-canary/ingress-canary-z6hh6" Oct 30 11:54:10.215380 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.215295 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-kubelet\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.215380 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.215306 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-trusted-ca-bundle\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:54:10.215380 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.215317 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-slash\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.215380 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.215369 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f2f3c38-f710-491a-9728-7c641291d679-data-volume\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:10.215582 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.215404 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-socket-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.215582 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.215478 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f2f3c38-f710-491a-9728-7c641291d679-data-volume\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:10.215582 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.215515 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d23976a-d44f-4b71-8b58-04932007cc17-cert\") pod \"ingress-canary-z6hh6\" (UID: \"9d23976a-d44f-4b71-8b58-04932007cc17\") " pod="openshift-ingress-canary/ingress-canary-z6hh6" Oct 30 11:54:10.232190 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.232161 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8ccv\" (UniqueName: \"kubernetes.io/projected/260d2659-42a8-461f-8994-0c5f9522a80d-kube-api-access-h8ccv\") pod \"network-metrics-daemon-rh7rw\" (UID: \"260d2659-42a8-461f-8994-0c5f9522a80d\") " pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:54:10.232302 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.232285 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8tg9\" (UniqueName: \"kubernetes.io/projected/31df51a8-4f22-4745-b25f-8b69654f2845-kube-api-access-v8tg9\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.232788 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.232768 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lswd\" (UniqueName: \"kubernetes.io/projected/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-kube-api-access-6lswd\") pod \"periodic-gathering-nr74p-m5v7z\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:54:10.232916 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.232895 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jhg2\" (UniqueName: \"kubernetes.io/projected/617427c7-590d-4d80-8c4e-e11c7812dca0-kube-api-access-5jhg2\") pod \"node-resolver-nhbdl\" (UID: \"617427c7-590d-4d80-8c4e-e11c7812dca0\") " pod="openshift-dns/node-resolver-nhbdl" Oct 30 11:54:10.233255 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.233236 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5g55\" (UniqueName: \"kubernetes.io/projected/d897418e-880a-46a9-8b13-9c2887bfddb7-kube-api-access-s5g55\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:54:10.234774 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.234758 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxpq6\" (UniqueName: \"kubernetes.io/projected/4f2f3c38-f710-491a-9728-7c641291d679-kube-api-access-nxpq6\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:10.234919 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.234904 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpwfh\" (UniqueName: \"kubernetes.io/projected/778f7f74-6134-47ab-a978-cf3462316e72-kube-api-access-xpwfh\") pod \"service-ca-5fffc97bdf-gqgxk\" (UID: \"778f7f74-6134-47ab-a978-cf3462316e72\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gqgxk" Oct 30 11:54:10.239222 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.239208 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l95t8\" (UniqueName: \"kubernetes.io/projected/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e-kube-api-access-l95t8\") pod \"node-ca-7c297\" (UID: \"29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e\") " pod="openshift-image-registry/node-ca-7c297" Oct 30 11:54:10.240199 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.240183 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jzbb\" (UniqueName: \"kubernetes.io/projected/363a0adc-193f-48d3-9628-1b9d10665597-kube-api-access-2jzbb\") pod \"network-check-target-ld8pr\" (UID: \"363a0adc-193f-48d3-9628-1b9d10665597\") " pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:54:10.244315 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.244300 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmctv\" (UniqueName: \"kubernetes.io/projected/1a948657-c616-435c-90db-b880a0db5bd9-kube-api-access-lmctv\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:54:10.244604 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.244589 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj598\" (UniqueName: \"kubernetes.io/projected/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-kube-api-access-hj598\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.245389 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.245369 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jtv5\" (UniqueName: \"kubernetes.io/projected/554f0b9b-bae8-478b-b3b1-85b376a79cc9-kube-api-access-7jtv5\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.249353 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.249333 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp7n7\" (UniqueName: \"kubernetes.io/projected/82a77175-bb28-474f-b4ec-dac73d790410-kube-api-access-sp7n7\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.249434 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.249367 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p7nr\" (UniqueName: \"kubernetes.io/projected/9d23976a-d44f-4b71-8b58-04932007cc17-kube-api-access-5p7nr\") pod \"ingress-canary-z6hh6\" (UID: \"9d23976a-d44f-4b71-8b58-04932007cc17\") " pod="openshift-ingress-canary/ingress-canary-z6hh6" Oct 30 11:54:10.250234 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.250216 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx8v8\" (UniqueName: \"kubernetes.io/projected/b2937779-b03d-4a5b-87bc-f062283eeb2e-kube-api-access-bx8v8\") pod \"iptables-alerter-htqhw\" (UID: \"b2937779-b03d-4a5b-87bc-f062283eeb2e\") " pod="openshift-network-operator/iptables-alerter-htqhw" Oct 30 11:54:10.315999 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.315921 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-etc-kubernetes\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.315999 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.315953 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-cni-netd\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.315999 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.315969 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-modprobe-d\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.316262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316006 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d897418e-880a-46a9-8b13-9c2887bfddb7-cnibin\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:54:10.316262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316028 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-lib-modules\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.316262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316050 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-etc-kubernetes\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316044 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-multus-cni-dir\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316106 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.316262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316112 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d897418e-880a-46a9-8b13-9c2887bfddb7-cnibin\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:54:10.316262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316122 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-sys-fs\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.316262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316132 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-multus-cni-dir\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316138 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-lib-modules\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.316262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316050 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-cni-netd\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316154 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-multus-socket-dir-parent\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316164 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-sys-fs\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.316262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316177 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-var-lib-kubelet\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316190 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-modprobe-d\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.316262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316203 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.316262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316218 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-var-lib-kubelet\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316219 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-multus-socket-dir-parent\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316249 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-kubelet\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316266 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-slash\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316281 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-socket-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316296 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-etc-selinux\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316299 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-kubelet\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316316 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-cnibin\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316329 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-slash\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316344 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-socket-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316345 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-hostroot\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316355 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-etc-selinux\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316364 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-cnibin\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316374 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e-host\") pod \"node-ca-7c297\" (UID: \"29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e\") " pod="openshift-image-registry/node-ca-7c297" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316379 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-hostroot\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316390 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46289f03-406c-4831-9ad1-0e6751fd7bd5-dbus\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316405 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-sysconfig\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316410 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e-host\") pod \"node-ca-7c297\" (UID: \"29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e\") " pod="openshift-image-registry/node-ca-7c297" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316418 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-run\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316433 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-run-openvswitch\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316456 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-run-netns\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316458 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-sysconfig\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316480 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46289f03-406c-4831-9ad1-0e6751fd7bd5-kubelet-config\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316486 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-run-netns\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316520 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46289f03-406c-4831-9ad1-0e6751fd7bd5-kubelet-config\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316506 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-run-systemd\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316541 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46289f03-406c-4831-9ad1-0e6751fd7bd5-dbus\") pod \"global-pull-secret-syncer-mqv9x\" (UID: \"46289f03-406c-4831-9ad1-0e6751fd7bd5\") " pod="kube-system/global-pull-secret-syncer-mqv9x" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316552 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-run\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316556 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-sysctl-conf\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316572 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-host\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316577 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-run-systemd\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316528 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-run-openvswitch\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316588 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-run-multus-certs\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316624 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d897418e-880a-46a9-8b13-9c2887bfddb7-system-cni-dir\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316647 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316657 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-host\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316669 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-run-multus-certs\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316674 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-var-lib-openvswitch\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316684 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d897418e-880a-46a9-8b13-9c2887bfddb7-system-cni-dir\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316702 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-sysctl-conf\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316706 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-var-lib-openvswitch\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316709 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316704 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-cni-bin\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316732 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-cni-bin\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316753 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-sys\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316779 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-var-lib-cni-multus\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316804 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-sys\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316803 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-multus-conf-dir\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316834 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-multus-conf-dir\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316834 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-run-ovn\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316856 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-var-lib-cni-multus\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316862 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-run-ovn\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316864 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-var-lib-kubelet\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316898 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-system-cni-dir\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316911 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-var-lib-kubelet\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316914 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-etc-openvswitch\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316931 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-etc-openvswitch\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316948 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-log-socket\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316969 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-run-ovn-kubernetes\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.316938 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316976 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-log-socket\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316979 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-system-cni-dir\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.316994 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/617427c7-590d-4d80-8c4e-e11c7812dca0-hosts-file\") pod \"node-resolver-nhbdl\" (UID: \"617427c7-590d-4d80-8c4e-e11c7812dca0\") " pod="openshift-dns/node-resolver-nhbdl" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317020 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d897418e-880a-46a9-8b13-9c2887bfddb7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317160 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs\") pod \"network-metrics-daemon-rh7rw\" (UID: \"260d2659-42a8-461f-8994-0c5f9522a80d\") " pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317188 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-node-log\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317218 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d897418e-880a-46a9-8b13-9c2887bfddb7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7bvp\" (UID: \"d897418e-880a-46a9-8b13-9c2887bfddb7\") " pod="openshift-multus/multus-additional-cni-plugins-d7bvp" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317233 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-node-log\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317234 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-run-ovn-kubernetes\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317200 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/617427c7-590d-4d80-8c4e-e11c7812dca0-hosts-file\") pod \"node-resolver-nhbdl\" (UID: \"617427c7-590d-4d80-8c4e-e11c7812dca0\") " pod="openshift-dns/node-resolver-nhbdl" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317237 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-kubernetes\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317301 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-kubernetes\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317323 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-run-k8s-cni-cncf-io\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317350 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317386 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2937779-b03d-4a5b-87bc-f062283eeb2e-host-slash\") pod \"iptables-alerter-htqhw\" (UID: \"b2937779-b03d-4a5b-87bc-f062283eeb2e\") " pod="openshift-network-operator/iptables-alerter-htqhw" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317411 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-registration-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317418 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-run-k8s-cni-cncf-io\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317435 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-device-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317461 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-var-lib-cni-bin\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317472 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2937779-b03d-4a5b-87bc-f062283eeb2e-host-slash\") pod \"iptables-alerter-htqhw\" (UID: \"b2937779-b03d-4a5b-87bc-f062283eeb2e\") " pod="openshift-network-operator/iptables-alerter-htqhw" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317489 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-systemd-units\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:10.317489 4468 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317517 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f2f3c38-f710-491a-9728-7c641291d679-crio-socket\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317523 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-device-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317548 4468 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317563 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31df51a8-4f22-4745-b25f-8b69654f2845-host-var-lib-cni-bin\") pod \"multus-6pxtt\" (UID: \"31df51a8-4f22-4745-b25f-8b69654f2845\") " pod="openshift-multus/multus-6pxtt" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317578 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f2f3c38-f710-491a-9728-7c641291d679-crio-socket\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317523 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82a77175-bb28-474f-b4ec-dac73d790410-registration-dir\") pod \"aws-ebs-csi-driver-node-z6lr6\" (UID: \"82a77175-bb28-474f-b4ec-dac73d790410\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z6lr6" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317580 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-systemd-units\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:10.317588 4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls podName:1a948657-c616-435c-90db-b880a0db5bd9 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:10.817567348 +0000 UTC m=+1.851705749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls") pod "dns-default-p4nvb" (UID: "1a948657-c616-435c-90db-b880a0db5bd9") : secret "dns-default-metrics-tls" not found Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317555 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:10.317626 4468 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317633 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-sysctl-d\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317657 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-run-netns\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:10.317675 4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls podName:4f2f3c38-f710-491a-9728-7c641291d679 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:10.817661258 +0000 UTC m=+1.851799632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dvql" (UID: "4f2f3c38-f710-491a-9728-7c641291d679") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317698 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/554f0b9b-bae8-478b-b3b1-85b376a79cc9-host-run-netns\") pod \"ovnkube-node-qq48w\" (UID: \"554f0b9b-bae8-478b-b3b1-85b376a79cc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317702 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-systemd\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317775 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-systemd\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.318773 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.317778 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11f0fad5-3b17-44fe-a2e0-4e99f496ab6e-etc-sysctl-d\") pod \"tuned-2jgjj\" (UID: \"11f0fad5-3b17-44fe-a2e0-4e99f496ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Oct 30 11:54:10.319782 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.319554 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260d2659-42a8-461f-8994-0c5f9522a80d-metrics-certs\") pod \"network-metrics-daemon-rh7rw\" (UID: \"260d2659-42a8-461f-8994-0c5f9522a80d\") " pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:54:10.384507 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.384479 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh7rw" Oct 30 11:54:10.710892 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.710866 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rh7rw"] Oct 30 11:54:10.712617 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:10.712591 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260d2659_42a8_461f_8994_0c5f9522a80d.slice/crio-9dcfdf7408b947fc61c1518fababd34b312a95992e94189223a16f2ca71637d0 WatchSource:0}: Error finding container 9dcfdf7408b947fc61c1518fababd34b312a95992e94189223a16f2ca71637d0: Status 404 returned error can't find the container with id 9dcfdf7408b947fc61c1518fababd34b312a95992e94189223a16f2ca71637d0 Oct 30 11:54:10.715223 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.715204 4468 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 11:54:10.821171 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.821129 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:54:10.821357 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.821189 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:10.821357 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:10.821299 4468 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:10.821357 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:10.821355 4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls podName:4f2f3c38-f710-491a-9728-7c641291d679 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:11.821342582 +0000 UTC m=+2.855480955 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dvql" (UID: "4f2f3c38-f710-491a-9728-7c641291d679") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:10.823502 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.823471 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a948657-c616-435c-90db-b880a0db5bd9-metrics-tls\") pod \"dns-default-p4nvb\" (UID: \"1a948657-c616-435c-90db-b880a0db5bd9\") " pod="openshift-dns/dns-default-p4nvb" Oct 30 11:54:10.988379 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:10.988344 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p4nvb" Oct 30 11:54:11.127381 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:11.127342 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rh7rw" event={"ID":"260d2659-42a8-461f-8994-0c5f9522a80d","Type":"ContainerStarted","Data":"9dcfdf7408b947fc61c1518fababd34b312a95992e94189223a16f2ca71637d0"} Oct 30 11:54:11.310827 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:11.310768 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p4nvb"] Oct 30 11:54:11.312961 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:11.312936 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a948657_c616_435c_90db_b880a0db5bd9.slice/crio-e8c000212bf059cf2e64d5d08ed5e31bfe41c16eafc632a3933f5b78cdb21e66 WatchSource:0}: Error finding container e8c000212bf059cf2e64d5d08ed5e31bfe41c16eafc632a3933f5b78cdb21e66: Status 404 returned error can't find the container with id e8c000212bf059cf2e64d5d08ed5e31bfe41c16eafc632a3933f5b78cdb21e66 Oct 30 11:54:11.829240 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:11.829166 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:11.829426 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:11.829331 4468 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:11.829493 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:11.829485 4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls podName:4f2f3c38-f710-491a-9728-7c641291d679 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:13.829464397 +0000 UTC m=+4.863602785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dvql" (UID: "4f2f3c38-f710-491a-9728-7c641291d679") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:12.132988 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:12.132891 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rh7rw" event={"ID":"260d2659-42a8-461f-8994-0c5f9522a80d","Type":"ContainerStarted","Data":"bebb86db82d60b1be712127acb75fadfcd9856e62ce027b839ac05e27c1cc8f5"} Oct 30 11:54:12.133572 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:12.133000 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rh7rw" event={"ID":"260d2659-42a8-461f-8994-0c5f9522a80d","Type":"ContainerStarted","Data":"3e961e2cdfdbe80f6e775291278fa110ad1e43e2057b2376f6316b3d44ecef53"} Oct 30 11:54:12.134400 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:12.134373 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p4nvb" event={"ID":"1a948657-c616-435c-90db-b880a0db5bd9","Type":"ContainerStarted","Data":"e8c000212bf059cf2e64d5d08ed5e31bfe41c16eafc632a3933f5b78cdb21e66"} Oct 30 11:54:12.956649 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:12.956620 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:54:13.139147 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:13.139053 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p4nvb" event={"ID":"1a948657-c616-435c-90db-b880a0db5bd9","Type":"ContainerStarted","Data":"f892b1622e4f16369167626b08451df99050ee41685d14d1206a24d22a32f398"} Oct 30 11:54:13.139147 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:13.139126 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p4nvb" event={"ID":"1a948657-c616-435c-90db-b880a0db5bd9","Type":"ContainerStarted","Data":"aae57c5ad928fd41b5fe0128b782ccf8b67a3527016127ee13fcb4fc0644813c"} Oct 30 11:54:13.842604 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:13.842567 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:13.842783 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:13.842707 4468 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:13.842783 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:13.842759 4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls podName:4f2f3c38-f710-491a-9728-7c641291d679 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:17.84274699 +0000 UTC m=+8.876885367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dvql" (UID: "4f2f3c38-f710-491a-9728-7c641291d679") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:14.142434 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:14.142354 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p4nvb" Oct 30 11:54:14.708903 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:14.708869 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:17.865283 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:17.865241 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:17.867602 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:17.867576 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f2f3c38-f710-491a-9728-7c641291d679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dvql\" (UID: \"4f2f3c38-f710-491a-9728-7c641291d679\") " pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:17.888482 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:17.888453 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6dvql" Oct 30 11:54:18.183710 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:18.183684 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6dvql"] Oct 30 11:54:18.186976 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:18.186950 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2f3c38_f710_491a_9728_7c641291d679.slice/crio-a2fa216e89ed44a6140cba4cc1761a4be4581facf0ee6bb02122539b9af0c46b WatchSource:0}: Error finding container a2fa216e89ed44a6140cba4cc1761a4be4581facf0ee6bb02122539b9af0c46b: Status 404 returned error can't find the container with id a2fa216e89ed44a6140cba4cc1761a4be4581facf0ee6bb02122539b9af0c46b Oct 30 11:54:18.493925 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:18.493849 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:18.494107 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:18.494092 4468 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 11:54:19.161653 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:19.161619 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6dvql" event={"ID":"4f2f3c38-f710-491a-9728-7c641291d679","Type":"ContainerStarted","Data":"e04c394bec0027993a3b8930e78017da619dcd9059303516c16cd945423c0c7e"} Oct 30 11:54:19.162103 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:19.161697 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6dvql" event={"ID":"4f2f3c38-f710-491a-9728-7c641291d679","Type":"ContainerStarted","Data":"1133c3a9bcee333fa0f20e6b5a8e77f7b5c579202604265bb8ec395e1c5c44d0"} Oct 30 11:54:19.162103 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:19.161718 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6dvql" event={"ID":"4f2f3c38-f710-491a-9728-7c641291d679","Type":"ContainerStarted","Data":"a2fa216e89ed44a6140cba4cc1761a4be4581facf0ee6bb02122539b9af0c46b"} Oct 30 11:54:20.166657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:20.166630 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6dvql" event={"ID":"4f2f3c38-f710-491a-9728-7c641291d679","Type":"ContainerStarted","Data":"6de5ec634b3684f3abe1d70191345062aec3227ec55da3f21600b372656a2019"} Oct 30 11:54:22.879364 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:22.879332 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7wdvn"] Oct 30 11:54:22.882827 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:22.882803 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7wdvn" Oct 30 11:54:22.917097 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:22.917043 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-l54hq\"" Oct 30 11:54:22.917273 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:22.917174 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 11:54:22.917658 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:22.917630 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7wdvn"] Oct 30 11:54:22.998254 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:22.998222 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8447c7c9c8-ptjpp"] Oct 30 11:54:23.001349 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.001315 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.005288 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.005256 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f6144c8e-3940-4027-a3b2-04792894d8bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7wdvn\" (UID: \"f6144c8e-3940-4027-a3b2-04792894d8bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7wdvn" Oct 30 11:54:23.016115 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.016093 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 11:54:23.016914 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.016891 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 11:54:23.016914 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.016894 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kv86z\"" Oct 30 11:54:23.027491 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.027377 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 11:54:23.040816 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.040791 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 11:54:23.041594 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.041569 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8447c7c9c8-ptjpp"] Oct 30 11:54:23.105746 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.105710 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ztmt\" (UniqueName: \"kubernetes.io/projected/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-kube-api-access-7ztmt\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.105928 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.105752 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-trusted-ca\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.105928 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.105789 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-image-registry-private-configuration\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.105928 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.105890 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-registry-tls\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.105928 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.105913 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-installation-pull-secrets\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.106162 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.105967 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-registry-certificates\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.106162 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.106005 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f6144c8e-3940-4027-a3b2-04792894d8bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7wdvn\" (UID: \"f6144c8e-3940-4027-a3b2-04792894d8bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7wdvn" Oct 30 11:54:23.106162 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.106078 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-ca-trust-extracted\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.106162 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.106111 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-bound-sa-token\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.108529 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.108508 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f6144c8e-3940-4027-a3b2-04792894d8bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7wdvn\" (UID: \"f6144c8e-3940-4027-a3b2-04792894d8bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7wdvn" Oct 30 11:54:23.203848 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.203819 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7wdvn" Oct 30 11:54:23.207128 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.207102 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-trusted-ca\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.207209 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.207148 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-image-registry-private-configuration\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.207209 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.207171 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-registry-tls\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.207408 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.207380 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-installation-pull-secrets\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.207483 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.207440 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-registry-certificates\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.207537 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.207494 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-ca-trust-extracted\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.207586 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.207521 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-bound-sa-token\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.207642 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.207584 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ztmt\" (UniqueName: \"kubernetes.io/projected/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-kube-api-access-7ztmt\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.208851 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.208186 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-ca-trust-extracted\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.208851 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.208257 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-trusted-ca\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.208851 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.208322 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-registry-certificates\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.209701 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.209670 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-image-registry-private-configuration\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.209801 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.209726 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-registry-tls\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.209801 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.209784 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-installation-pull-secrets\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.235025 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.234997 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ztmt\" (UniqueName: \"kubernetes.io/projected/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-kube-api-access-7ztmt\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.235629 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.235610 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9884f855-dc2f-4dc0-abbe-7ddf956f4cd5-bound-sa-token\") pod \"image-registry-8447c7c9c8-ptjpp\" (UID: \"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5\") " pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.317919 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.317886 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:23.504794 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.504769 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7wdvn"] Oct 30 11:54:23.506434 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:23.506410 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6144c8e_3940_4027_a3b2_04792894d8bc.slice/crio-3a4bb0292a598496dee4ffa92e1ef58b121947483ed0c19b8a224538b2820b70 WatchSource:0}: Error finding container 3a4bb0292a598496dee4ffa92e1ef58b121947483ed0c19b8a224538b2820b70: Status 404 returned error can't find the container with id 3a4bb0292a598496dee4ffa92e1ef58b121947483ed0c19b8a224538b2820b70 Oct 30 11:54:23.596245 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:23.596209 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 11:54:23.615708 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:23.615683 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8447c7c9c8-ptjpp"] Oct 30 11:54:23.616701 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:23.616677 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9884f855_dc2f_4dc0_abbe_7ddf956f4cd5.slice/crio-c6f3aea048500881089dd02f86dcb2da6b9180e637a98b78605eb34fed1699c1 WatchSource:0}: Error finding container c6f3aea048500881089dd02f86dcb2da6b9180e637a98b78605eb34fed1699c1: Status 404 returned error can't find the container with id c6f3aea048500881089dd02f86dcb2da6b9180e637a98b78605eb34fed1699c1 Oct 30 11:54:24.147423 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:24.147395 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p4nvb" Oct 30 11:54:24.179864 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:24.179824 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" event={"ID":"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5","Type":"ContainerStarted","Data":"e3a6415f59b96b4c83ee90170eda30c8ce8f084ec410f00665c3ef4bdd559e4c"} Oct 30 11:54:24.180012 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:24.179886 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:24.180012 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:24.179907 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" event={"ID":"9884f855-dc2f-4dc0-abbe-7ddf956f4cd5","Type":"ContainerStarted","Data":"c6f3aea048500881089dd02f86dcb2da6b9180e637a98b78605eb34fed1699c1"} Oct 30 11:54:24.181117 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:24.181065 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7wdvn" event={"ID":"f6144c8e-3940-4027-a3b2-04792894d8bc","Type":"ContainerStarted","Data":"3a4bb0292a598496dee4ffa92e1ef58b121947483ed0c19b8a224538b2820b70"} Oct 30 11:54:24.238312 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:24.238258 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" podStartSLOduration=2.23824372 podStartE2EDuration="2.23824372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:24.237664113 +0000 UTC m=+15.271802509" watchObservedRunningTime="2026-10-30 11:54:24.23824372 +0000 UTC m=+15.272382116" Oct 30 11:54:25.186417 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.186375 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7wdvn" event={"ID":"f6144c8e-3940-4027-a3b2-04792894d8bc","Type":"ContainerStarted","Data":"ff5a371496770b74c58bc78bd86108a251d4306a369cae7d20549788d44e5551"} Oct 30 11:54:25.186869 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.186541 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7wdvn" Oct 30 11:54:25.191219 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.191200 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7wdvn" Oct 30 11:54:25.314542 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.314481 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7wdvn" podStartSLOduration=2.061099265 podStartE2EDuration="3.314463133s" totalImagesPullingTime="1.253363868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:25.270656542 +0000 UTC m=+16.304794937" watchObservedRunningTime="2026-10-30 11:54:25.314463133 +0000 UTC m=+16.348601530" Oct 30 11:54:25.581929 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.581856 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8"] Oct 30 11:54:25.585052 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.585036 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8" Oct 30 11:54:25.611535 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.611504 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-h6q2l\"" Oct 30 11:54:25.611535 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.611513 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 11:54:25.611718 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.611564 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 11:54:25.611718 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.611646 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 11:54:25.612442 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.612424 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8"] Oct 30 11:54:25.700308 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.700274 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 11:54:25.700470 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.700363 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 11:54:25.726228 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.726204 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffa08fe5-b4b9-4775-9d46-486c0065135a-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-h8pv8\" (UID: \"ffa08fe5-b4b9-4775-9d46-486c0065135a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8" Oct 30 11:54:25.726272 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.726242 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkzf7\" (UniqueName: \"kubernetes.io/projected/ffa08fe5-b4b9-4775-9d46-486c0065135a-kube-api-access-zkzf7\") pod \"prometheus-operator-9cd89b74c-h8pv8\" (UID: \"ffa08fe5-b4b9-4775-9d46-486c0065135a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8" Oct 30 11:54:25.726272 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.726261 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa08fe5-b4b9-4775-9d46-486c0065135a-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-h8pv8\" (UID: \"ffa08fe5-b4b9-4775-9d46-486c0065135a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8" Oct 30 11:54:25.726379 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.726364 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ffa08fe5-b4b9-4775-9d46-486c0065135a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-h8pv8\" (UID: \"ffa08fe5-b4b9-4775-9d46-486c0065135a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8" Oct 30 11:54:25.827586 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.827543 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkzf7\" (UniqueName: \"kubernetes.io/projected/ffa08fe5-b4b9-4775-9d46-486c0065135a-kube-api-access-zkzf7\") pod \"prometheus-operator-9cd89b74c-h8pv8\" (UID: \"ffa08fe5-b4b9-4775-9d46-486c0065135a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8" Oct 30 11:54:25.827586 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.827589 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa08fe5-b4b9-4775-9d46-486c0065135a-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-h8pv8\" (UID: \"ffa08fe5-b4b9-4775-9d46-486c0065135a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8" Oct 30 11:54:25.827801 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.827627 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ffa08fe5-b4b9-4775-9d46-486c0065135a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-h8pv8\" (UID: \"ffa08fe5-b4b9-4775-9d46-486c0065135a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8" Oct 30 11:54:25.827801 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.827676 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffa08fe5-b4b9-4775-9d46-486c0065135a-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-h8pv8\" (UID: \"ffa08fe5-b4b9-4775-9d46-486c0065135a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8" Oct 30 11:54:25.828384 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.828359 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffa08fe5-b4b9-4775-9d46-486c0065135a-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-h8pv8\" (UID: \"ffa08fe5-b4b9-4775-9d46-486c0065135a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8" Oct 30 11:54:25.830042 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.830016 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ffa08fe5-b4b9-4775-9d46-486c0065135a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-h8pv8\" (UID: \"ffa08fe5-b4b9-4775-9d46-486c0065135a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8" Oct 30 11:54:25.830138 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.830085 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffa08fe5-b4b9-4775-9d46-486c0065135a-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-h8pv8\" (UID: \"ffa08fe5-b4b9-4775-9d46-486c0065135a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8" Oct 30 11:54:25.850352 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.850299 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkzf7\" (UniqueName: \"kubernetes.io/projected/ffa08fe5-b4b9-4775-9d46-486c0065135a-kube-api-access-zkzf7\") pod \"prometheus-operator-9cd89b74c-h8pv8\" (UID: \"ffa08fe5-b4b9-4775-9d46-486c0065135a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8" Oct 30 11:54:25.897908 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:25.897874 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8" Oct 30 11:54:26.190902 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:26.190882 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8"] Oct 30 11:54:26.192326 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:26.192300 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa08fe5_b4b9_4775_9d46_486c0065135a.slice/crio-87bcf93e4b03dc020879b1fbd9ac5c9cd1b69292df8a3f6bc9e20a68becd49ed WatchSource:0}: Error finding container 87bcf93e4b03dc020879b1fbd9ac5c9cd1b69292df8a3f6bc9e20a68becd49ed: Status 404 returned error can't find the container with id 87bcf93e4b03dc020879b1fbd9ac5c9cd1b69292df8a3f6bc9e20a68becd49ed Oct 30 11:54:27.195082 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:27.195034 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8" event={"ID":"ffa08fe5-b4b9-4775-9d46-486c0065135a","Type":"ContainerStarted","Data":"87bcf93e4b03dc020879b1fbd9ac5c9cd1b69292df8a3f6bc9e20a68becd49ed"} Oct 30 11:54:28.199770 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:28.199734 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8" event={"ID":"ffa08fe5-b4b9-4775-9d46-486c0065135a","Type":"ContainerStarted","Data":"105d77b28482702b3001f6088dc3d7f93135f6d8cb87b446630f1d6f4d84f618"} Oct 30 11:54:28.200364 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:28.199792 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8" event={"ID":"ffa08fe5-b4b9-4775-9d46-486c0065135a","Type":"ContainerStarted","Data":"cafe3da8efb1362404a46c2f81f816298554058005964f83c5b748873bb92ec9"} Oct 30 11:54:28.240580 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:28.240531 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-h8pv8" podStartSLOduration=2.024594224 podStartE2EDuration="3.240516833s" totalImagesPullingTime="1.215922609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:28.240062102 +0000 UTC m=+19.274200497" watchObservedRunningTime="2026-10-30 11:54:28.240516833 +0000 UTC m=+19.274655225" Oct 30 11:54:30.291513 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.291481 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c"] Oct 30 11:54:30.296191 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.296175 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c" Oct 30 11:54:30.314104 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.314054 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 11:54:30.325878 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.325853 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-582gs\"" Oct 30 11:54:30.326672 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.326647 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c"] Oct 30 11:54:30.327907 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.327886 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m2dqw"] Oct 30 11:54:30.331909 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.331889 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.352012 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.351988 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 11:54:30.414588 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.414562 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xwvts\"" Oct 30 11:54:30.414841 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.414829 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 11:54:30.414969 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.414954 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 11:54:30.415804 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.415792 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 11:54:30.458170 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.458139 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bf169833-c766-4b19-a563-092d490f0090-root\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.458170 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.458171 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlgld\" (UniqueName: \"kubernetes.io/projected/bf169833-c766-4b19-a563-092d490f0090-kube-api-access-nlgld\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.458356 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.458203 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa996c80-1b20-4275-b2ce-26bb5b7d08e0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wrq8c\" (UID: \"aa996c80-1b20-4275-b2ce-26bb5b7d08e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c" Oct 30 11:54:30.458356 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.458336 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf169833-c766-4b19-a563-092d490f0090-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.458427 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.458371 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf169833-c766-4b19-a563-092d490f0090-sys\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.458464 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.458424 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm2dg\" (UniqueName: \"kubernetes.io/projected/aa996c80-1b20-4275-b2ce-26bb5b7d08e0-kube-api-access-wm2dg\") pod \"openshift-state-metrics-7bd7489957-wrq8c\" (UID: \"aa996c80-1b20-4275-b2ce-26bb5b7d08e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c" Oct 30 11:54:30.458464 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.458446 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa996c80-1b20-4275-b2ce-26bb5b7d08e0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wrq8c\" (UID: \"aa996c80-1b20-4275-b2ce-26bb5b7d08e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c" Oct 30 11:54:30.458526 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.458468 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bf169833-c766-4b19-a563-092d490f0090-node-exporter-tls\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.458526 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.458498 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bf169833-c766-4b19-a563-092d490f0090-node-exporter-wtmp\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.458587 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.458557 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa996c80-1b20-4275-b2ce-26bb5b7d08e0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wrq8c\" (UID: \"aa996c80-1b20-4275-b2ce-26bb5b7d08e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c" Oct 30 11:54:30.458587 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.458574 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bf169833-c766-4b19-a563-092d490f0090-node-exporter-accelerators-collector-config\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.458645 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.458593 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bf169833-c766-4b19-a563-092d490f0090-node-exporter-textfile\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.458645 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.458624 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf169833-c766-4b19-a563-092d490f0090-metrics-client-ca\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.559802 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.559711 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf169833-c766-4b19-a563-092d490f0090-sys\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.559802 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.559767 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm2dg\" (UniqueName: \"kubernetes.io/projected/aa996c80-1b20-4275-b2ce-26bb5b7d08e0-kube-api-access-wm2dg\") pod \"openshift-state-metrics-7bd7489957-wrq8c\" (UID: \"aa996c80-1b20-4275-b2ce-26bb5b7d08e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c" Oct 30 11:54:30.559802 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.559794 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa996c80-1b20-4275-b2ce-26bb5b7d08e0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wrq8c\" (UID: \"aa996c80-1b20-4275-b2ce-26bb5b7d08e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c" Oct 30 11:54:30.560039 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.559823 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bf169833-c766-4b19-a563-092d490f0090-node-exporter-tls\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.560039 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.559825 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf169833-c766-4b19-a563-092d490f0090-sys\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.560039 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.559840 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bf169833-c766-4b19-a563-092d490f0090-node-exporter-wtmp\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.560039 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.559886 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa996c80-1b20-4275-b2ce-26bb5b7d08e0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wrq8c\" (UID: \"aa996c80-1b20-4275-b2ce-26bb5b7d08e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c" Oct 30 11:54:30.560039 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.559917 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bf169833-c766-4b19-a563-092d490f0090-node-exporter-accelerators-collector-config\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.560039 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.559944 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bf169833-c766-4b19-a563-092d490f0090-node-exporter-textfile\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.560039 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.559964 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bf169833-c766-4b19-a563-092d490f0090-node-exporter-wtmp\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.560039 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.559981 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf169833-c766-4b19-a563-092d490f0090-metrics-client-ca\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.560039 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.560016 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bf169833-c766-4b19-a563-092d490f0090-root\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.560039 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.560040 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlgld\" (UniqueName: \"kubernetes.io/projected/bf169833-c766-4b19-a563-092d490f0090-kube-api-access-nlgld\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.560510 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.560094 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa996c80-1b20-4275-b2ce-26bb5b7d08e0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wrq8c\" (UID: \"aa996c80-1b20-4275-b2ce-26bb5b7d08e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c" Oct 30 11:54:30.560510 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.560128 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf169833-c766-4b19-a563-092d490f0090-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.560799 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.560768 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bf169833-c766-4b19-a563-092d490f0090-node-exporter-textfile\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.560887 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.560865 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bf169833-c766-4b19-a563-092d490f0090-root\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.561122 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.561100 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa996c80-1b20-4275-b2ce-26bb5b7d08e0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wrq8c\" (UID: \"aa996c80-1b20-4275-b2ce-26bb5b7d08e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c" Oct 30 11:54:30.561587 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.561352 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf169833-c766-4b19-a563-092d490f0090-metrics-client-ca\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.561587 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.561362 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bf169833-c766-4b19-a563-092d490f0090-node-exporter-accelerators-collector-config\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.563783 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.563755 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bf169833-c766-4b19-a563-092d490f0090-node-exporter-tls\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.563935 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.563881 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa996c80-1b20-4275-b2ce-26bb5b7d08e0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wrq8c\" (UID: \"aa996c80-1b20-4275-b2ce-26bb5b7d08e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c" Oct 30 11:54:30.565723 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.564528 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf169833-c766-4b19-a563-092d490f0090-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.566298 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.566276 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa996c80-1b20-4275-b2ce-26bb5b7d08e0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wrq8c\" (UID: \"aa996c80-1b20-4275-b2ce-26bb5b7d08e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c" Oct 30 11:54:30.612955 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.612927 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm2dg\" (UniqueName: \"kubernetes.io/projected/aa996c80-1b20-4275-b2ce-26bb5b7d08e0-kube-api-access-wm2dg\") pod \"openshift-state-metrics-7bd7489957-wrq8c\" (UID: \"aa996c80-1b20-4275-b2ce-26bb5b7d08e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c" Oct 30 11:54:30.613167 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.613148 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlgld\" (UniqueName: \"kubernetes.io/projected/bf169833-c766-4b19-a563-092d490f0090-kube-api-access-nlgld\") pod \"node-exporter-m2dqw\" (UID: \"bf169833-c766-4b19-a563-092d490f0090\") " pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.648203 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.648175 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m2dqw" Oct 30 11:54:30.659698 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:30.659668 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf169833_c766_4b19_a563_092d490f0090.slice/crio-a1923be51a18830edc63b25286123bae75aed780572e3807165abfb61379828e WatchSource:0}: Error finding container a1923be51a18830edc63b25286123bae75aed780572e3807165abfb61379828e: Status 404 returned error can't find the container with id a1923be51a18830edc63b25286123bae75aed780572e3807165abfb61379828e Oct 30 11:54:30.911743 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:30.911717 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c" Oct 30 11:54:31.211961 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:31.211886 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m2dqw" event={"ID":"bf169833-c766-4b19-a563-092d490f0090","Type":"ContainerStarted","Data":"a1923be51a18830edc63b25286123bae75aed780572e3807165abfb61379828e"} Oct 30 11:54:31.229402 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:31.229374 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c"] Oct 30 11:54:31.232985 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:31.232960 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa996c80_1b20_4275_b2ce_26bb5b7d08e0.slice/crio-696d850ff9b8e23f776462c282cbb3c2cf44f6026da390c0466e5fdecfacd1b1 WatchSource:0}: Error finding container 696d850ff9b8e23f776462c282cbb3c2cf44f6026da390c0466e5fdecfacd1b1: Status 404 returned error can't find the container with id 696d850ff9b8e23f776462c282cbb3c2cf44f6026da390c0466e5fdecfacd1b1 Oct 30 11:54:32.217988 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:32.217956 4468 generic.go:356] "Generic (PLEG): container finished" podID="bf169833-c766-4b19-a563-092d490f0090" containerID="bd975aba107044ebdfbd0f1206a68697b709d26144922941c5201789ddc03b9a" exitCode=0 Oct 30 11:54:32.218476 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:32.218052 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m2dqw" event={"ID":"bf169833-c766-4b19-a563-092d490f0090","Type":"ContainerDied","Data":"bd975aba107044ebdfbd0f1206a68697b709d26144922941c5201789ddc03b9a"} Oct 30 11:54:32.220197 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:32.220175 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c" event={"ID":"aa996c80-1b20-4275-b2ce-26bb5b7d08e0","Type":"ContainerStarted","Data":"637e4e84d7e0f54a7a16390bd42e4c1984a7e7de3aff0a138cba2c23dda53773"} Oct 30 11:54:32.220303 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:32.220226 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c" event={"ID":"aa996c80-1b20-4275-b2ce-26bb5b7d08e0","Type":"ContainerStarted","Data":"4a014f19345fb8836c89d48fdeed603dd88c75d07020ce793c5672584193c0cf"} Oct 30 11:54:32.220303 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:32.220257 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c" event={"ID":"aa996c80-1b20-4275-b2ce-26bb5b7d08e0","Type":"ContainerStarted","Data":"696d850ff9b8e23f776462c282cbb3c2cf44f6026da390c0466e5fdecfacd1b1"} Oct 30 11:54:32.731104 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:32.731060 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qq48w" Oct 30 11:54:33.224904 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:33.224868 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m2dqw" event={"ID":"bf169833-c766-4b19-a563-092d490f0090","Type":"ContainerStarted","Data":"d8770eca7c2a7aad982ce528f841a0728584c551d67413a8e958eac0de04c1c3"} Oct 30 11:54:33.225370 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:33.224924 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m2dqw" event={"ID":"bf169833-c766-4b19-a563-092d490f0090","Type":"ContainerStarted","Data":"7d2e3042cead50d93c33bd5438932de0564cdf5eedc092f46a144c23e4764867"} Oct 30 11:54:33.226863 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:33.226841 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c" event={"ID":"aa996c80-1b20-4275-b2ce-26bb5b7d08e0","Type":"ContainerStarted","Data":"578d1d8673b7c8facbfb306b0b2cf8f4ee5c0f47e5ef990e3a4cef463b1378ed"} Oct 30 11:54:33.270887 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:33.270841 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m2dqw" podStartSLOduration=2.273969733 podStartE2EDuration="3.270826293s" totalImagesPullingTime="930.67897ms" totalInitContainerRuntime="66.17759ms" isStatefulPod=true podCreationTimestamp="2026-10-30 11:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:33.270332208 +0000 UTC m=+24.304470603" watchObservedRunningTime="2026-10-30 11:54:33.270826293 +0000 UTC m=+24.304964689" Oct 30 11:54:33.334505 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:33.334457 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrq8c" podStartSLOduration=2.11363261 podStartE2EDuration="3.334439164s" totalImagesPullingTime="1.220806554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:33.333697109 +0000 UTC m=+24.367835501" watchObservedRunningTime="2026-10-30 11:54:33.334439164 +0000 UTC m=+24.368577560" Oct 30 11:54:37.855737 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:37.855704 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qjvts" Oct 30 11:54:40.757533 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:40.757496 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-4mb5r"] Oct 30 11:54:40.790248 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:40.790215 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-4mb5r"] Oct 30 11:54:40.790399 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:40.790339 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-4mb5r" Oct 30 11:54:40.805508 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:40.805487 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 11:54:40.816510 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:40.816487 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 11:54:40.816510 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:40.816504 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-9skpx\"" Oct 30 11:54:40.829232 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:40.829210 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24aac159-d3bb-4ed2-8132-93456b41cefe-tmp\") pod \"downloads-6dd789c8b7-4mb5r\" (UID: \"24aac159-d3bb-4ed2-8132-93456b41cefe\") " pod="openshift-console/downloads-6dd789c8b7-4mb5r" Oct 30 11:54:40.829395 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:40.829256 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsxm2\" (UniqueName: \"kubernetes.io/projected/24aac159-d3bb-4ed2-8132-93456b41cefe-kube-api-access-rsxm2\") pod \"downloads-6dd789c8b7-4mb5r\" (UID: \"24aac159-d3bb-4ed2-8132-93456b41cefe\") " pod="openshift-console/downloads-6dd789c8b7-4mb5r" Oct 30 11:54:40.930352 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:40.930318 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsxm2\" (UniqueName: \"kubernetes.io/projected/24aac159-d3bb-4ed2-8132-93456b41cefe-kube-api-access-rsxm2\") pod \"downloads-6dd789c8b7-4mb5r\" (UID: \"24aac159-d3bb-4ed2-8132-93456b41cefe\") " pod="openshift-console/downloads-6dd789c8b7-4mb5r" Oct 30 11:54:40.930487 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:40.930370 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24aac159-d3bb-4ed2-8132-93456b41cefe-tmp\") pod \"downloads-6dd789c8b7-4mb5r\" (UID: \"24aac159-d3bb-4ed2-8132-93456b41cefe\") " pod="openshift-console/downloads-6dd789c8b7-4mb5r" Oct 30 11:54:40.930676 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:40.930661 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24aac159-d3bb-4ed2-8132-93456b41cefe-tmp\") pod \"downloads-6dd789c8b7-4mb5r\" (UID: \"24aac159-d3bb-4ed2-8132-93456b41cefe\") " pod="openshift-console/downloads-6dd789c8b7-4mb5r" Oct 30 11:54:40.955935 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:40.955909 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsxm2\" (UniqueName: \"kubernetes.io/projected/24aac159-d3bb-4ed2-8132-93456b41cefe-kube-api-access-rsxm2\") pod \"downloads-6dd789c8b7-4mb5r\" (UID: \"24aac159-d3bb-4ed2-8132-93456b41cefe\") " pod="openshift-console/downloads-6dd789c8b7-4mb5r" Oct 30 11:54:41.114588 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:41.114513 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-4mb5r" Oct 30 11:54:41.450898 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:41.450820 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-4mb5r"] Oct 30 11:54:41.454398 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:41.454371 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24aac159_d3bb_4ed2_8132_93456b41cefe.slice/crio-3559ae60eefde97cbb75ae22f35edd06132aef56df678ae08b8884c936a05bd9 WatchSource:0}: Error finding container 3559ae60eefde97cbb75ae22f35edd06132aef56df678ae08b8884c936a05bd9: Status 404 returned error can't find the container with id 3559ae60eefde97cbb75ae22f35edd06132aef56df678ae08b8884c936a05bd9 Oct 30 11:54:42.263356 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:42.263316 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-4mb5r" event={"ID":"24aac159-d3bb-4ed2-8132-93456b41cefe","Type":"ContainerStarted","Data":"3559ae60eefde97cbb75ae22f35edd06132aef56df678ae08b8884c936a05bd9"} Oct 30 11:54:42.959401 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:42.959373 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ld8pr" Oct 30 11:54:45.191973 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:45.191946 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8447c7c9c8-ptjpp" Oct 30 11:54:51.032556 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.032466 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-746b465c4c-f2wrf"] Oct 30 11:54:51.038624 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.038604 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.073196 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.073168 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 11:54:51.073196 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.073195 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 11:54:51.073406 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.073261 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 11:54:51.077229 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.077205 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-746b465c4c-f2wrf"] Oct 30 11:54:51.077368 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.077356 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 11:54:51.077368 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.077357 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 11:54:51.084981 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.084961 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q22rn\"" Oct 30 11:54:51.117269 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.117232 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-serving-cert\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.117446 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.117277 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/502e4e7f-b7fc-4445-8fa3-4689fc236621-tmp\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.117446 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.117308 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-service-ca\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.117446 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.117379 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-config\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.117603 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.117470 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-oauth-serving-cert\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.117603 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.117517 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29v87\" (UniqueName: \"kubernetes.io/projected/502e4e7f-b7fc-4445-8fa3-4689fc236621-kube-api-access-29v87\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.117671 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.117624 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-oauth-config\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.218478 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.218442 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-oauth-config\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.218959 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.218504 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-serving-cert\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.218959 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.218604 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/502e4e7f-b7fc-4445-8fa3-4689fc236621-tmp\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.218959 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.218635 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-service-ca\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.218959 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.218740 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-config\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.218959 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.218781 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-oauth-serving-cert\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.218959 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.218837 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29v87\" (UniqueName: \"kubernetes.io/projected/502e4e7f-b7fc-4445-8fa3-4689fc236621-kube-api-access-29v87\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.219392 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.219092 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/502e4e7f-b7fc-4445-8fa3-4689fc236621-tmp\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.219671 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.219649 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-oauth-serving-cert\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.220115 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.220097 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-service-ca\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.220321 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.220300 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-config\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.221303 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.221285 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-oauth-config\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.221682 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.221653 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-serving-cert\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.244509 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.244486 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29v87\" (UniqueName: \"kubernetes.io/projected/502e4e7f-b7fc-4445-8fa3-4689fc236621-kube-api-access-29v87\") pod \"console-746b465c4c-f2wrf\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.350942 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.350866 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:54:51.659513 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:51.659483 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-746b465c4c-f2wrf"] Oct 30 11:54:51.661213 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:54:51.661184 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502e4e7f_b7fc_4445_8fa3_4689fc236621.slice/crio-62db16e4cb49b2de92a2a80f7002ab12404e62444246ffa39bdfbfea28af3dcf WatchSource:0}: Error finding container 62db16e4cb49b2de92a2a80f7002ab12404e62444246ffa39bdfbfea28af3dcf: Status 404 returned error can't find the container with id 62db16e4cb49b2de92a2a80f7002ab12404e62444246ffa39bdfbfea28af3dcf Oct 30 11:54:52.304322 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:52.304278 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746b465c4c-f2wrf" event={"ID":"502e4e7f-b7fc-4445-8fa3-4689fc236621","Type":"ContainerStarted","Data":"62db16e4cb49b2de92a2a80f7002ab12404e62444246ffa39bdfbfea28af3dcf"} Oct 30 11:54:53.602332 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:54:53.602289 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 11:54:55.318305 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:55.318266 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746b465c4c-f2wrf" event={"ID":"502e4e7f-b7fc-4445-8fa3-4689fc236621","Type":"ContainerStarted","Data":"0bea793a46ceddad76ec3c123c38f83a768cd16707bc4ade6a9ec72904d09625"} Oct 30 11:54:55.361899 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:55.361835 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-746b465c4c-f2wrf" podStartSLOduration=1.420374094 podStartE2EDuration="4.361816313s" totalImagesPullingTime="2.941442219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:55.360164706 +0000 UTC m=+46.394303106" watchObservedRunningTime="2026-10-30 11:54:55.361816313 +0000 UTC m=+46.395954711" Oct 30 11:54:57.327608 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:57.327574 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-4mb5r" event={"ID":"24aac159-d3bb-4ed2-8132-93456b41cefe","Type":"ContainerStarted","Data":"b0f01f87cb803d83cd63ff57488eab8eb0aa8d102175fe480a7e8bfddc4252c4"} Oct 30 11:54:57.328130 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:57.327848 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-4mb5r" Oct 30 11:54:57.329484 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:57.329453 4468 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-4mb5r container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.15:8080/\": dial tcp 10.134.0.15:8080: connect: connection refused" start-of-body= Oct 30 11:54:57.329590 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:57.329510 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-4mb5r" podUID="24aac159-d3bb-4ed2-8132-93456b41cefe" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.15:8080/\": dial tcp 10.134.0.15:8080: connect: connection refused" Oct 30 11:54:57.398269 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:57.398174 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-4mb5r" podStartSLOduration=1.70422367 podStartE2EDuration="17.398152908s" totalImagesPullingTime="15.693929238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:57.386273562 +0000 UTC m=+48.420411958" watchObservedRunningTime="2026-10-30 11:54:57.398152908 +0000 UTC m=+48.432291305" Oct 30 11:54:58.345218 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:54:58.345181 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-4mb5r" Oct 30 11:55:01.351236 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.351199 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:55:01.351691 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.351258 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:55:01.356810 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.356786 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:55:01.646910 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.646833 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f8fbd69dd-ldt79"] Oct 30 11:55:01.678782 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.678750 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f8fbd69dd-ldt79"] Oct 30 11:55:01.678981 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.678889 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.749795 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.749762 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 11:55:01.814012 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.813974 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-trusted-ca-bundle\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.814209 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.814031 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3194a1d6-33a8-41b9-a5ac-546f306ed90d-tmp\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.814209 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.814059 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-serving-cert\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.814209 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.814175 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-oauth-serving-cert\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.814384 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.814225 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vldld\" (UniqueName: \"kubernetes.io/projected/3194a1d6-33a8-41b9-a5ac-546f306ed90d-kube-api-access-vldld\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.814384 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.814331 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-oauth-config\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.814490 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.814394 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-service-ca\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.814490 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.814434 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-config\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.915174 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.915142 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3194a1d6-33a8-41b9-a5ac-546f306ed90d-tmp\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.915374 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.915188 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-serving-cert\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.915374 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.915215 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-oauth-serving-cert\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.915374 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.915250 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vldld\" (UniqueName: \"kubernetes.io/projected/3194a1d6-33a8-41b9-a5ac-546f306ed90d-kube-api-access-vldld\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.915374 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.915275 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-oauth-config\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.915374 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.915293 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-service-ca\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.915374 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.915327 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-config\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.915374 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.915371 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-trusted-ca-bundle\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.915748 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.915612 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3194a1d6-33a8-41b9-a5ac-546f306ed90d-tmp\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.916215 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.916187 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-oauth-serving-cert\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.916324 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.916231 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-config\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.916432 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.916410 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-trusted-ca-bundle\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.916491 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.916440 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-service-ca\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.918924 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.918900 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-oauth-config\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.920426 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.920407 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-serving-cert\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:01.939601 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:01.939570 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vldld\" (UniqueName: \"kubernetes.io/projected/3194a1d6-33a8-41b9-a5ac-546f306ed90d-kube-api-access-vldld\") pod \"console-7f8fbd69dd-ldt79\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:02.009415 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:02.009374 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:02.352521 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:02.352491 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:55:02.355349 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:02.354213 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f8fbd69dd-ldt79"] Oct 30 11:55:02.356792 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:55:02.356764 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3194a1d6_33a8_41b9_a5ac_546f306ed90d.slice/crio-713c16ef7c08f5e4687a00950b8d5dee6883adc5fee0d53be2798ed96f32faef WatchSource:0}: Error finding container 713c16ef7c08f5e4687a00950b8d5dee6883adc5fee0d53be2798ed96f32faef: Status 404 returned error can't find the container with id 713c16ef7c08f5e4687a00950b8d5dee6883adc5fee0d53be2798ed96f32faef Oct 30 11:55:03.352968 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:03.352925 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f8fbd69dd-ldt79" event={"ID":"3194a1d6-33a8-41b9-a5ac-546f306ed90d","Type":"ContainerStarted","Data":"70afcf1fcd61132b841747c12427cf7502a1d14e32ac4aeff79e28693bb8d643"} Oct 30 11:55:03.353548 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:03.352989 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f8fbd69dd-ldt79" event={"ID":"3194a1d6-33a8-41b9-a5ac-546f306ed90d","Type":"ContainerStarted","Data":"713c16ef7c08f5e4687a00950b8d5dee6883adc5fee0d53be2798ed96f32faef"} Oct 30 11:55:03.432188 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:03.432132 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7f8fbd69dd-ldt79" podStartSLOduration=2.432117945 podStartE2EDuration="2.432117945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:03.419620277 +0000 UTC m=+54.453758674" watchObservedRunningTime="2026-10-30 11:55:03.432117945 +0000 UTC m=+54.466256342" Oct 30 11:55:12.009757 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:12.009716 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:12.010329 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:12.009782 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:12.014743 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:12.014718 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:12.392781 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:12.392703 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:12.472835 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:12.472806 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-746b465c4c-f2wrf"] Oct 30 11:55:15.329442 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.329407 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-nr74p-m5v7z"] Oct 30 11:55:15.329870 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.329594 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-nr74p-m5v7z" podUID="cf0f7768-5ba5-4b50-a601-0c4202b2b66e" containerName="insights-gathering" containerID="cri-o://4e44951acbbd226653848fc81fd977d4c9e6d45583cef8d14e912a75d4f07d02" gracePeriod=30 Oct 30 11:55:15.515217 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.515187 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-f4rss-rqldf"] Oct 30 11:55:15.531797 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.531770 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4rss-rqldf" Oct 30 11:55:15.537616 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.537550 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-f4rss-rqldf"] Oct 30 11:55:15.631905 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.631879 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5c7a481-35d8-48ce-a22d-cc72d04111f2-trusted-ca-bundle\") pod \"periodic-gathering-f4rss-rqldf\" (UID: \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\") " pod="openshift-insights/periodic-gathering-f4rss-rqldf" Oct 30 11:55:15.632047 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.631912 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a5c7a481-35d8-48ce-a22d-cc72d04111f2-archives-path\") pod \"periodic-gathering-f4rss-rqldf\" (UID: \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\") " pod="openshift-insights/periodic-gathering-f4rss-rqldf" Oct 30 11:55:15.632047 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.631939 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dj4r\" (UniqueName: \"kubernetes.io/projected/a5c7a481-35d8-48ce-a22d-cc72d04111f2-kube-api-access-2dj4r\") pod \"periodic-gathering-f4rss-rqldf\" (UID: \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\") " pod="openshift-insights/periodic-gathering-f4rss-rqldf" Oct 30 11:55:15.632152 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.632105 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5c7a481-35d8-48ce-a22d-cc72d04111f2-service-ca-bundle\") pod \"periodic-gathering-f4rss-rqldf\" (UID: \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\") " pod="openshift-insights/periodic-gathering-f4rss-rqldf" Oct 30 11:55:15.642709 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.642686 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:55:15.732612 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.732578 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-archives-path\") pod \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " Oct 30 11:55:15.732793 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.732634 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-kube-api-access-6lswd\" (UniqueName: \"kubernetes.io/projected/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-kube-api-access-6lswd\") pod \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " Oct 30 11:55:15.732793 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.732658 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-service-ca-bundle\") pod \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " Oct 30 11:55:15.732793 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.732681 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-trusted-ca-bundle\") pod \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\" (UID: \"cf0f7768-5ba5-4b50-a601-0c4202b2b66e\") " Oct 30 11:55:15.732793 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.732776 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5c7a481-35d8-48ce-a22d-cc72d04111f2-service-ca-bundle\") pod \"periodic-gathering-f4rss-rqldf\" (UID: \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\") " pod="openshift-insights/periodic-gathering-f4rss-rqldf" Oct 30 11:55:15.733016 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.732833 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5c7a481-35d8-48ce-a22d-cc72d04111f2-trusted-ca-bundle\") pod \"periodic-gathering-f4rss-rqldf\" (UID: \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\") " pod="openshift-insights/periodic-gathering-f4rss-rqldf" Oct 30 11:55:15.733016 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.732855 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a5c7a481-35d8-48ce-a22d-cc72d04111f2-archives-path\") pod \"periodic-gathering-f4rss-rqldf\" (UID: \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\") " pod="openshift-insights/periodic-gathering-f4rss-rqldf" Oct 30 11:55:15.733016 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.732890 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dj4r\" (UniqueName: \"kubernetes.io/projected/a5c7a481-35d8-48ce-a22d-cc72d04111f2-kube-api-access-2dj4r\") pod \"periodic-gathering-f4rss-rqldf\" (UID: \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\") " pod="openshift-insights/periodic-gathering-f4rss-rqldf" Oct 30 11:55:15.733016 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.732932 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-archives-path" pod "cf0f7768-5ba5-4b50-a601-0c4202b2b66e" (UID: "cf0f7768-5ba5-4b50-a601-0c4202b2b66e"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:55:15.733250 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.733114 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-service-ca-bundle" pod "cf0f7768-5ba5-4b50-a601-0c4202b2b66e" (UID: "cf0f7768-5ba5-4b50-a601-0c4202b2b66e"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:15.733304 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.733250 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-trusted-ca-bundle" pod "cf0f7768-5ba5-4b50-a601-0c4202b2b66e" (UID: "cf0f7768-5ba5-4b50-a601-0c4202b2b66e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:15.733361 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.733339 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a5c7a481-35d8-48ce-a22d-cc72d04111f2-archives-path\") pod \"periodic-gathering-f4rss-rqldf\" (UID: \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\") " pod="openshift-insights/periodic-gathering-f4rss-rqldf" Oct 30 11:55:15.733644 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.733624 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5c7a481-35d8-48ce-a22d-cc72d04111f2-service-ca-bundle\") pod \"periodic-gathering-f4rss-rqldf\" (UID: \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\") " pod="openshift-insights/periodic-gathering-f4rss-rqldf" Oct 30 11:55:15.733815 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.733794 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5c7a481-35d8-48ce-a22d-cc72d04111f2-trusted-ca-bundle\") pod \"periodic-gathering-f4rss-rqldf\" (UID: \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\") " pod="openshift-insights/periodic-gathering-f4rss-rqldf" Oct 30 11:55:15.735253 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.735229 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-kube-api-access-6lswd" pod "cf0f7768-5ba5-4b50-a601-0c4202b2b66e" (UID: "cf0f7768-5ba5-4b50-a601-0c4202b2b66e"). InnerVolumeSpecName "kube-api-access-6lswd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:55:15.756083 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.756040 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dj4r\" (UniqueName: \"kubernetes.io/projected/a5c7a481-35d8-48ce-a22d-cc72d04111f2-kube-api-access-2dj4r\") pod \"periodic-gathering-f4rss-rqldf\" (UID: \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\") " pod="openshift-insights/periodic-gathering-f4rss-rqldf" Oct 30 11:55:15.833738 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.833703 4468 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-service-ca-bundle\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:55:15.833738 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.833732 4468 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-trusted-ca-bundle\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:55:15.833738 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.833742 4468 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-archives-path\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:55:15.833934 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.833751 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lswd\" (UniqueName: \"kubernetes.io/projected/cf0f7768-5ba5-4b50-a601-0c4202b2b66e-kube-api-access-6lswd\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:55:15.860766 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:15.860742 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4rss-rqldf" Oct 30 11:55:16.177711 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.177684 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-f4rss-rqldf"] Oct 30 11:55:16.179232 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:55:16.179203 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c7a481_35d8_48ce_a22d_cc72d04111f2.slice/crio-3c89954c23d099f9b9b6b7c377e1bc7b869fd6335fc5acdec46e5d1eb11c4606 WatchSource:0}: Error finding container 3c89954c23d099f9b9b6b7c377e1bc7b869fd6335fc5acdec46e5d1eb11c4606: Status 404 returned error can't find the container with id 3c89954c23d099f9b9b6b7c377e1bc7b869fd6335fc5acdec46e5d1eb11c4606 Oct 30 11:55:16.277192 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.277157 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-77449bbc64-2hhck"] Oct 30 11:55:16.277451 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.277435 4468 cpu_manager.go:414] "containerMap: removing container" podUID="cf0f7768-5ba5-4b50-a601-0c4202b2b66e" containerName="insights-gathering" Oct 30 11:55:16.277493 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.277452 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf0f7768-5ba5-4b50-a601-0c4202b2b66e" containerName="insights-gathering" Oct 30 11:55:16.277531 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.277522 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf0f7768-5ba5-4b50-a601-0c4202b2b66e" containerName="insights-gathering" Oct 30 11:55:16.279194 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.279179 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.303327 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.303298 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77449bbc64-2hhck"] Oct 30 11:55:16.405010 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.404972 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4rss-rqldf" event={"ID":"a5c7a481-35d8-48ce-a22d-cc72d04111f2","Type":"ContainerStarted","Data":"a2dce22859851bb12a2177648249382c07d97926924a3e5c1f9ff7125a3f3685"} Oct 30 11:55:16.405507 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.405031 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4rss-rqldf" event={"ID":"a5c7a481-35d8-48ce-a22d-cc72d04111f2","Type":"ContainerStarted","Data":"3c89954c23d099f9b9b6b7c377e1bc7b869fd6335fc5acdec46e5d1eb11c4606"} Oct 30 11:55:16.410939 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.406682 4468 generic.go:356] "Generic (PLEG): container finished" podID="cf0f7768-5ba5-4b50-a601-0c4202b2b66e" containerID="4e44951acbbd226653848fc81fd977d4c9e6d45583cef8d14e912a75d4f07d02" exitCode=2 Oct 30 11:55:16.410939 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.406751 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nr74p-m5v7z" event={"ID":"cf0f7768-5ba5-4b50-a601-0c4202b2b66e","Type":"ContainerDied","Data":"4e44951acbbd226653848fc81fd977d4c9e6d45583cef8d14e912a75d4f07d02"} Oct 30 11:55:16.410939 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.406786 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nr74p-m5v7z" event={"ID":"cf0f7768-5ba5-4b50-a601-0c4202b2b66e","Type":"ContainerDied","Data":"3545637cdbb817797ebdd7286acb317ed11a1a4a4daf67cc766df1471b9a3c50"} Oct 30 11:55:16.410939 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.406811 4468 scope.go:122] "RemoveContainer" containerID="4e44951acbbd226653848fc81fd977d4c9e6d45583cef8d14e912a75d4f07d02" Oct 30 11:55:16.410939 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.406936 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nr74p-m5v7z" Oct 30 11:55:16.418767 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.418747 4468 scope.go:122] "RemoveContainer" containerID="4e44951acbbd226653848fc81fd977d4c9e6d45583cef8d14e912a75d4f07d02" Oct 30 11:55:16.419093 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:55:16.419043 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e44951acbbd226653848fc81fd977d4c9e6d45583cef8d14e912a75d4f07d02\": container with ID starting with 4e44951acbbd226653848fc81fd977d4c9e6d45583cef8d14e912a75d4f07d02 not found: ID does not exist" containerID="4e44951acbbd226653848fc81fd977d4c9e6d45583cef8d14e912a75d4f07d02" Oct 30 11:55:16.419200 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.419123 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e44951acbbd226653848fc81fd977d4c9e6d45583cef8d14e912a75d4f07d02"} err="failed to get container status \"4e44951acbbd226653848fc81fd977d4c9e6d45583cef8d14e912a75d4f07d02\": rpc error: code = NotFound desc = could not find container \"4e44951acbbd226653848fc81fd977d4c9e6d45583cef8d14e912a75d4f07d02\": container with ID starting with 4e44951acbbd226653848fc81fd977d4c9e6d45583cef8d14e912a75d4f07d02 not found: ID does not exist" Oct 30 11:55:16.437526 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.437450 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dt57\" (UniqueName: \"kubernetes.io/projected/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-kube-api-access-8dt57\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.437526 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.437487 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-console-serving-cert\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.437526 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.437509 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-tmp\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.437766 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.437582 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-console-oauth-config\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.437766 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.437657 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-console-config\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.437766 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.437696 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-trusted-ca-bundle\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.437766 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.437715 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-service-ca\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.437766 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.437748 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-oauth-serving-cert\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.444177 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.444125 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-f4rss-rqldf" podStartSLOduration=1.444113124 podStartE2EDuration="1.444113124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:16.443761611 +0000 UTC m=+67.477900007" watchObservedRunningTime="2026-10-30 11:55:16.444113124 +0000 UTC m=+67.478251519" Oct 30 11:55:16.478101 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.478055 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-nr74p-m5v7z"] Oct 30 11:55:16.486434 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.483328 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-nr74p-m5v7z"] Oct 30 11:55:16.538475 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.538437 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-tmp\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.538475 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.538486 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-console-oauth-config\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.538720 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.538519 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-console-config\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.538720 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.538560 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-trusted-ca-bundle\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.538720 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.538585 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-service-ca\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.538720 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.538605 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-oauth-serving-cert\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.538720 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.538710 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dt57\" (UniqueName: \"kubernetes.io/projected/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-kube-api-access-8dt57\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.538974 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.538744 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-console-serving-cert\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.538974 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.538960 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-tmp\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.539353 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.539331 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-service-ca\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.539494 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.539466 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-oauth-serving-cert\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.539608 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.539507 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-trusted-ca-bundle\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.539818 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.539799 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-console-config\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.541176 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.541150 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-console-oauth-config\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.541275 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.541258 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-console-serving-cert\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.562591 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.562559 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dt57\" (UniqueName: \"kubernetes.io/projected/6b6c009e-76fe-4449-88d2-4f4d65d44b7b-kube-api-access-8dt57\") pod \"console-77449bbc64-2hhck\" (UID: \"6b6c009e-76fe-4449-88d2-4f4d65d44b7b\") " pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.591449 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.591417 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:16.900950 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:16.900925 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77449bbc64-2hhck"] Oct 30 11:55:16.902719 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:55:16.902691 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6c009e_76fe_4449_88d2_4f4d65d44b7b.slice/crio-87b195106bde09bd6b5743114a1a086b45849870b894a4f12dc9f906da25c21f WatchSource:0}: Error finding container 87b195106bde09bd6b5743114a1a086b45849870b894a4f12dc9f906da25c21f: Status 404 returned error can't find the container with id 87b195106bde09bd6b5743114a1a086b45849870b894a4f12dc9f906da25c21f Oct 30 11:55:17.087031 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:17.086994 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf0f7768-5ba5-4b50-a601-0c4202b2b66e" path="/var/lib/kubelet/pods/cf0f7768-5ba5-4b50-a601-0c4202b2b66e/volumes" Oct 30 11:55:17.412709 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:17.412675 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77449bbc64-2hhck" event={"ID":"6b6c009e-76fe-4449-88d2-4f4d65d44b7b","Type":"ContainerStarted","Data":"0b2b0e440ac28201115ab10cc241e5e8cf8f431312d3e6a99aa531469e524fc1"} Oct 30 11:55:17.413114 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:17.412720 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77449bbc64-2hhck" event={"ID":"6b6c009e-76fe-4449-88d2-4f4d65d44b7b","Type":"ContainerStarted","Data":"87b195106bde09bd6b5743114a1a086b45849870b894a4f12dc9f906da25c21f"} Oct 30 11:55:17.454837 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:17.454788 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-77449bbc64-2hhck" podStartSLOduration=1.454774458 podStartE2EDuration="1.454774458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:17.454141154 +0000 UTC m=+68.488279550" watchObservedRunningTime="2026-10-30 11:55:17.454774458 +0000 UTC m=+68.488912854" Oct 30 11:55:23.596961 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:55:23.596920 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 11:55:26.592061 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:26.592027 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:26.592608 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:26.592092 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:26.596634 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:26.596612 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:27.449657 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:27.449631 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77449bbc64-2hhck" Oct 30 11:55:27.526375 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:27.526348 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f8fbd69dd-ldt79"] Oct 30 11:55:37.492546 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.492502 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-746b465c4c-f2wrf" podUID="502e4e7f-b7fc-4445-8fa3-4689fc236621" containerName="console" containerID="cri-o://0bea793a46ceddad76ec3c123c38f83a768cd16707bc4ade6a9ec72904d09625" gracePeriod=15 Oct 30 11:55:37.763213 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.763148 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:55:37.809257 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.809228 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-serving-cert\") pod \"502e4e7f-b7fc-4445-8fa3-4689fc236621\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " Oct 30 11:55:37.809413 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.809269 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/502e4e7f-b7fc-4445-8fa3-4689fc236621-tmp\" (UniqueName: \"kubernetes.io/empty-dir/502e4e7f-b7fc-4445-8fa3-4689fc236621-tmp\") pod \"502e4e7f-b7fc-4445-8fa3-4689fc236621\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " Oct 30 11:55:37.809413 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.809286 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-oauth-config\") pod \"502e4e7f-b7fc-4445-8fa3-4689fc236621\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " Oct 30 11:55:37.809413 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.809362 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-config\" (UniqueName: \"kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-config\") pod \"502e4e7f-b7fc-4445-8fa3-4689fc236621\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " Oct 30 11:55:37.809413 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.809409 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/502e4e7f-b7fc-4445-8fa3-4689fc236621-kube-api-access-29v87\" (UniqueName: \"kubernetes.io/projected/502e4e7f-b7fc-4445-8fa3-4689fc236621-kube-api-access-29v87\") pod \"502e4e7f-b7fc-4445-8fa3-4689fc236621\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " Oct 30 11:55:37.809621 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.809448 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-oauth-serving-cert\") pod \"502e4e7f-b7fc-4445-8fa3-4689fc236621\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " Oct 30 11:55:37.809621 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.809482 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-service-ca\" (UniqueName: \"kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-service-ca\") pod \"502e4e7f-b7fc-4445-8fa3-4689fc236621\" (UID: \"502e4e7f-b7fc-4445-8fa3-4689fc236621\") " Oct 30 11:55:37.809621 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.809543 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/502e4e7f-b7fc-4445-8fa3-4689fc236621-tmp" pod "502e4e7f-b7fc-4445-8fa3-4689fc236621" (UID: "502e4e7f-b7fc-4445-8fa3-4689fc236621"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:55:37.809778 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.809697 4468 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/502e4e7f-b7fc-4445-8fa3-4689fc236621-tmp\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:55:37.809946 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.809900 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-oauth-serving-cert" pod "502e4e7f-b7fc-4445-8fa3-4689fc236621" (UID: "502e4e7f-b7fc-4445-8fa3-4689fc236621"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:37.810106 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.810004 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-service-ca" pod "502e4e7f-b7fc-4445-8fa3-4689fc236621" (UID: "502e4e7f-b7fc-4445-8fa3-4689fc236621"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:37.810106 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.810022 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-config" pod "502e4e7f-b7fc-4445-8fa3-4689fc236621" (UID: "502e4e7f-b7fc-4445-8fa3-4689fc236621"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:37.811961 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.811934 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/502e4e7f-b7fc-4445-8fa3-4689fc236621-kube-api-access-29v87" pod "502e4e7f-b7fc-4445-8fa3-4689fc236621" (UID: "502e4e7f-b7fc-4445-8fa3-4689fc236621"). InnerVolumeSpecName "kube-api-access-29v87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:55:37.811961 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.811953 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-oauth-config" pod "502e4e7f-b7fc-4445-8fa3-4689fc236621" (UID: "502e4e7f-b7fc-4445-8fa3-4689fc236621"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:37.812288 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.812265 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-serving-cert" pod "502e4e7f-b7fc-4445-8fa3-4689fc236621" (UID: "502e4e7f-b7fc-4445-8fa3-4689fc236621"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:37.910213 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.910182 4468 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-config\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:55:37.910213 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.910207 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29v87\" (UniqueName: \"kubernetes.io/projected/502e4e7f-b7fc-4445-8fa3-4689fc236621-kube-api-access-29v87\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:55:37.910213 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.910218 4468 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-oauth-serving-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:55:37.910431 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.910227 4468 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/502e4e7f-b7fc-4445-8fa3-4689fc236621-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:55:37.910431 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.910236 4468 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-serving-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:55:37.910431 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:37.910244 4468 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/502e4e7f-b7fc-4445-8fa3-4689fc236621-console-oauth-config\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:55:38.485266 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:38.485230 4468 generic.go:356] "Generic (PLEG): container finished" podID="502e4e7f-b7fc-4445-8fa3-4689fc236621" containerID="0bea793a46ceddad76ec3c123c38f83a768cd16707bc4ade6a9ec72904d09625" exitCode=0 Oct 30 11:55:38.485450 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:38.485276 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746b465c4c-f2wrf" Oct 30 11:55:38.485450 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:38.485275 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746b465c4c-f2wrf" event={"ID":"502e4e7f-b7fc-4445-8fa3-4689fc236621","Type":"ContainerDied","Data":"0bea793a46ceddad76ec3c123c38f83a768cd16707bc4ade6a9ec72904d09625"} Oct 30 11:55:38.485450 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:38.485399 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746b465c4c-f2wrf" event={"ID":"502e4e7f-b7fc-4445-8fa3-4689fc236621","Type":"ContainerDied","Data":"62db16e4cb49b2de92a2a80f7002ab12404e62444246ffa39bdfbfea28af3dcf"} Oct 30 11:55:38.485450 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:38.485422 4468 scope.go:122] "RemoveContainer" containerID="0bea793a46ceddad76ec3c123c38f83a768cd16707bc4ade6a9ec72904d09625" Oct 30 11:55:38.493797 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:38.493613 4468 scope.go:122] "RemoveContainer" containerID="0bea793a46ceddad76ec3c123c38f83a768cd16707bc4ade6a9ec72904d09625" Oct 30 11:55:38.494039 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:55:38.493883 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bea793a46ceddad76ec3c123c38f83a768cd16707bc4ade6a9ec72904d09625\": container with ID starting with 0bea793a46ceddad76ec3c123c38f83a768cd16707bc4ade6a9ec72904d09625 not found: ID does not exist" containerID="0bea793a46ceddad76ec3c123c38f83a768cd16707bc4ade6a9ec72904d09625" Oct 30 11:55:38.494039 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:38.493909 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bea793a46ceddad76ec3c123c38f83a768cd16707bc4ade6a9ec72904d09625"} err="failed to get container status \"0bea793a46ceddad76ec3c123c38f83a768cd16707bc4ade6a9ec72904d09625\": rpc error: code = NotFound desc = could not find container \"0bea793a46ceddad76ec3c123c38f83a768cd16707bc4ade6a9ec72904d09625\": container with ID starting with 0bea793a46ceddad76ec3c123c38f83a768cd16707bc4ade6a9ec72904d09625 not found: ID does not exist" Oct 30 11:55:38.534733 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:38.534699 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-746b465c4c-f2wrf"] Oct 30 11:55:38.546285 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:38.546259 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-746b465c4c-f2wrf"] Oct 30 11:55:39.086859 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:39.086824 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="502e4e7f-b7fc-4445-8fa3-4689fc236621" path="/var/lib/kubelet/pods/502e4e7f-b7fc-4445-8fa3-4689fc236621/volumes" Oct 30 11:55:52.546242 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:52.546175 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7f8fbd69dd-ldt79" podUID="3194a1d6-33a8-41b9-a5ac-546f306ed90d" containerName="console" containerID="cri-o://70afcf1fcd61132b841747c12427cf7502a1d14e32ac4aeff79e28693bb8d643" gracePeriod=15 Oct 30 11:55:52.794086 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:52.794054 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:52.925985 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:52.925956 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3194a1d6-33a8-41b9-a5ac-546f306ed90d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3194a1d6-33a8-41b9-a5ac-546f306ed90d-tmp\") pod \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " Oct 30 11:55:52.926178 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:52.926002 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-config\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-config\") pod \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " Oct 30 11:55:52.926178 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:52.926036 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-service-ca\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-service-ca\") pod \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " Oct 30 11:55:52.926178 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:52.926091 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-oauth-config\") pod \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " Oct 30 11:55:52.926178 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:52.926121 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-serving-cert\") pod \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " Oct 30 11:55:52.926178 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:52.926153 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-oauth-serving-cert\") pod \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " Oct 30 11:55:52.926411 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:52.926206 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-trusted-ca-bundle\") pod \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " Oct 30 11:55:52.926411 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:52.926274 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3194a1d6-33a8-41b9-a5ac-546f306ed90d-kube-api-access-vldld\" (UniqueName: \"kubernetes.io/projected/3194a1d6-33a8-41b9-a5ac-546f306ed90d-kube-api-access-vldld\") pod \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\" (UID: \"3194a1d6-33a8-41b9-a5ac-546f306ed90d\") " Oct 30 11:55:52.926411 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:52.926323 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3194a1d6-33a8-41b9-a5ac-546f306ed90d-tmp" pod "3194a1d6-33a8-41b9-a5ac-546f306ed90d" (UID: "3194a1d6-33a8-41b9-a5ac-546f306ed90d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:55:52.926576 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:52.926505 4468 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3194a1d6-33a8-41b9-a5ac-546f306ed90d-tmp\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:55:52.926632 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:52.926579 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-config" pod "3194a1d6-33a8-41b9-a5ac-546f306ed90d" (UID: "3194a1d6-33a8-41b9-a5ac-546f306ed90d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:52.926754 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:52.926721 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-service-ca" pod "3194a1d6-33a8-41b9-a5ac-546f306ed90d" (UID: "3194a1d6-33a8-41b9-a5ac-546f306ed90d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:52.926819 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:52.926729 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-oauth-serving-cert" pod "3194a1d6-33a8-41b9-a5ac-546f306ed90d" (UID: "3194a1d6-33a8-41b9-a5ac-546f306ed90d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:52.926819 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:52.926795 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-trusted-ca-bundle" pod "3194a1d6-33a8-41b9-a5ac-546f306ed90d" (UID: "3194a1d6-33a8-41b9-a5ac-546f306ed90d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:52.928293 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:52.928265 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-oauth-config" pod "3194a1d6-33a8-41b9-a5ac-546f306ed90d" (UID: "3194a1d6-33a8-41b9-a5ac-546f306ed90d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:52.928467 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:52.928448 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-serving-cert" pod "3194a1d6-33a8-41b9-a5ac-546f306ed90d" (UID: "3194a1d6-33a8-41b9-a5ac-546f306ed90d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:52.928547 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:52.928528 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3194a1d6-33a8-41b9-a5ac-546f306ed90d-kube-api-access-vldld" pod "3194a1d6-33a8-41b9-a5ac-546f306ed90d" (UID: "3194a1d6-33a8-41b9-a5ac-546f306ed90d"). InnerVolumeSpecName "kube-api-access-vldld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:55:53.026857 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:53.026807 4468 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-config\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:55:53.026857 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:53.026850 4468 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:55:53.026857 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:53.026859 4468 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-oauth-config\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:55:53.026857 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:53.026868 4468 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3194a1d6-33a8-41b9-a5ac-546f306ed90d-console-serving-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:55:53.026857 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:53.026877 4468 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-oauth-serving-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:55:53.027179 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:53.026885 4468 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3194a1d6-33a8-41b9-a5ac-546f306ed90d-trusted-ca-bundle\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:55:53.027179 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:53.026894 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vldld\" (UniqueName: \"kubernetes.io/projected/3194a1d6-33a8-41b9-a5ac-546f306ed90d-kube-api-access-vldld\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:55:53.537457 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:53.537421 4468 generic.go:356] "Generic (PLEG): container finished" podID="3194a1d6-33a8-41b9-a5ac-546f306ed90d" containerID="70afcf1fcd61132b841747c12427cf7502a1d14e32ac4aeff79e28693bb8d643" exitCode=0 Oct 30 11:55:53.537642 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:53.537474 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f8fbd69dd-ldt79" Oct 30 11:55:53.537642 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:53.537489 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f8fbd69dd-ldt79" event={"ID":"3194a1d6-33a8-41b9-a5ac-546f306ed90d","Type":"ContainerDied","Data":"70afcf1fcd61132b841747c12427cf7502a1d14e32ac4aeff79e28693bb8d643"} Oct 30 11:55:53.537642 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:53.537539 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f8fbd69dd-ldt79" event={"ID":"3194a1d6-33a8-41b9-a5ac-546f306ed90d","Type":"ContainerDied","Data":"713c16ef7c08f5e4687a00950b8d5dee6883adc5fee0d53be2798ed96f32faef"} Oct 30 11:55:53.537642 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:53.537565 4468 scope.go:122] "RemoveContainer" containerID="70afcf1fcd61132b841747c12427cf7502a1d14e32ac4aeff79e28693bb8d643" Oct 30 11:55:53.545871 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:53.545798 4468 scope.go:122] "RemoveContainer" containerID="70afcf1fcd61132b841747c12427cf7502a1d14e32ac4aeff79e28693bb8d643" Oct 30 11:55:53.546153 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:55:53.546129 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70afcf1fcd61132b841747c12427cf7502a1d14e32ac4aeff79e28693bb8d643\": container with ID starting with 70afcf1fcd61132b841747c12427cf7502a1d14e32ac4aeff79e28693bb8d643 not found: ID does not exist" containerID="70afcf1fcd61132b841747c12427cf7502a1d14e32ac4aeff79e28693bb8d643" Oct 30 11:55:53.546253 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:53.546163 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70afcf1fcd61132b841747c12427cf7502a1d14e32ac4aeff79e28693bb8d643"} err="failed to get container status \"70afcf1fcd61132b841747c12427cf7502a1d14e32ac4aeff79e28693bb8d643\": rpc error: code = NotFound desc = could not find container \"70afcf1fcd61132b841747c12427cf7502a1d14e32ac4aeff79e28693bb8d643\": container with ID starting with 70afcf1fcd61132b841747c12427cf7502a1d14e32ac4aeff79e28693bb8d643 not found: ID does not exist" Oct 30 11:55:53.587085 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:53.587046 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f8fbd69dd-ldt79"] Oct 30 11:55:53.597120 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:55:53.597058 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 11:55:53.598561 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:53.598542 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f8fbd69dd-ldt79"] Oct 30 11:55:55.087159 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:55:55.087130 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3194a1d6-33a8-41b9-a5ac-546f306ed90d" path="/var/lib/kubelet/pods/3194a1d6-33a8-41b9-a5ac-546f306ed90d/volumes" Oct 30 11:56:23.596497 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:56:23.596422 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 11:56:39.688708 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:56:39.688669 4468 generic.go:356] "Generic (PLEG): container finished" podID="a5c7a481-35d8-48ce-a22d-cc72d04111f2" containerID="a2dce22859851bb12a2177648249382c07d97926924a3e5c1f9ff7125a3f3685" exitCode=0 Oct 30 11:56:39.689231 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:56:39.688732 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4rss-rqldf" event={"ID":"a5c7a481-35d8-48ce-a22d-cc72d04111f2","Type":"ContainerDied","Data":"a2dce22859851bb12a2177648249382c07d97926924a3e5c1f9ff7125a3f3685"} Oct 30 11:56:40.822589 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:56:40.822564 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4rss-rqldf" Oct 30 11:56:40.884763 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:56:40.884732 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c7a481-35d8-48ce-a22d-cc72d04111f2-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5c7a481-35d8-48ce-a22d-cc72d04111f2-service-ca-bundle\") pod \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\" (UID: \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\") " Oct 30 11:56:40.884946 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:56:40.884778 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c7a481-35d8-48ce-a22d-cc72d04111f2-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a5c7a481-35d8-48ce-a22d-cc72d04111f2-archives-path\") pod \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\" (UID: \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\") " Oct 30 11:56:40.884946 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:56:40.884804 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c7a481-35d8-48ce-a22d-cc72d04111f2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5c7a481-35d8-48ce-a22d-cc72d04111f2-trusted-ca-bundle\") pod \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\" (UID: \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\") " Oct 30 11:56:40.884946 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:56:40.884829 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c7a481-35d8-48ce-a22d-cc72d04111f2-kube-api-access-2dj4r\" (UniqueName: \"kubernetes.io/projected/a5c7a481-35d8-48ce-a22d-cc72d04111f2-kube-api-access-2dj4r\") pod \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\" (UID: \"a5c7a481-35d8-48ce-a22d-cc72d04111f2\") " Oct 30 11:56:40.885138 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:56:40.885105 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c7a481-35d8-48ce-a22d-cc72d04111f2-archives-path" pod "a5c7a481-35d8-48ce-a22d-cc72d04111f2" (UID: "a5c7a481-35d8-48ce-a22d-cc72d04111f2"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:56:40.885200 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:56:40.885161 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c7a481-35d8-48ce-a22d-cc72d04111f2-service-ca-bundle" pod "a5c7a481-35d8-48ce-a22d-cc72d04111f2" (UID: "a5c7a481-35d8-48ce-a22d-cc72d04111f2"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:56:40.885200 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:56:40.885175 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c7a481-35d8-48ce-a22d-cc72d04111f2-trusted-ca-bundle" pod "a5c7a481-35d8-48ce-a22d-cc72d04111f2" (UID: "a5c7a481-35d8-48ce-a22d-cc72d04111f2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:56:40.886889 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:56:40.886861 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c7a481-35d8-48ce-a22d-cc72d04111f2-kube-api-access-2dj4r" pod "a5c7a481-35d8-48ce-a22d-cc72d04111f2" (UID: "a5c7a481-35d8-48ce-a22d-cc72d04111f2"). InnerVolumeSpecName "kube-api-access-2dj4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:56:40.985335 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:56:40.985255 4468 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5c7a481-35d8-48ce-a22d-cc72d04111f2-service-ca-bundle\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:56:40.985335 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:56:40.985285 4468 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a5c7a481-35d8-48ce-a22d-cc72d04111f2-archives-path\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:56:40.985335 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:56:40.985295 4468 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5c7a481-35d8-48ce-a22d-cc72d04111f2-trusted-ca-bundle\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:56:40.985335 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:56:40.985304 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dj4r\" (UniqueName: \"kubernetes.io/projected/a5c7a481-35d8-48ce-a22d-cc72d04111f2-kube-api-access-2dj4r\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:56:41.697414 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:56:41.697381 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4rss-rqldf" event={"ID":"a5c7a481-35d8-48ce-a22d-cc72d04111f2","Type":"ContainerDied","Data":"3c89954c23d099f9b9b6b7c377e1bc7b869fd6335fc5acdec46e5d1eb11c4606"} Oct 30 11:56:41.697414 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:56:41.697415 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4rss-rqldf" Oct 30 11:56:41.697624 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:56:41.697428 4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c89954c23d099f9b9b6b7c377e1bc7b869fd6335fc5acdec46e5d1eb11c4606" Oct 30 11:56:53.596699 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:56:53.596662 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 11:57:23.598341 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:57:23.598300 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 11:57:32.633988 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.633953 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8"] Oct 30 11:57:32.634473 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.634217 4468 cpu_manager.go:414] "containerMap: removing container" podUID="502e4e7f-b7fc-4445-8fa3-4689fc236621" containerName="console" Oct 30 11:57:32.634473 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.634229 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="502e4e7f-b7fc-4445-8fa3-4689fc236621" containerName="console" Oct 30 11:57:32.634473 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.634238 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3194a1d6-33a8-41b9-a5ac-546f306ed90d" containerName="console" Oct 30 11:57:32.634473 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.634244 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3194a1d6-33a8-41b9-a5ac-546f306ed90d" containerName="console" Oct 30 11:57:32.634473 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.634254 4468 cpu_manager.go:414] "containerMap: removing container" podUID="a5c7a481-35d8-48ce-a22d-cc72d04111f2" containerName="insights-gathering" Oct 30 11:57:32.634473 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.634259 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c7a481-35d8-48ce-a22d-cc72d04111f2" containerName="insights-gathering" Oct 30 11:57:32.634473 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.634309 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="502e4e7f-b7fc-4445-8fa3-4689fc236621" containerName="console" Oct 30 11:57:32.634473 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.634315 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3194a1d6-33a8-41b9-a5ac-546f306ed90d" containerName="console" Oct 30 11:57:32.634473 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.634321 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c7a481-35d8-48ce-a22d-cc72d04111f2" containerName="insights-gathering" Oct 30 11:57:32.637111 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.637095 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.656325 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.656274 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 11:57:32.656325 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.656280 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 11:57:32.656325 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.656307 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 11:57:32.656518 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.656346 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 11:57:32.667555 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.667530 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 11:57:32.667555 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.667554 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qvjrm\"" Oct 30 11:57:32.667771 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.667555 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 11:57:32.668668 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.668645 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8"] Oct 30 11:57:32.762521 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.762494 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.762521 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.762523 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.762711 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.762548 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.762711 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.762618 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.762711 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.762665 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.762806 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.762735 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6z6h\" (UniqueName: \"kubernetes.io/projected/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-kube-api-access-v6z6h\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.762806 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.762775 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.863370 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.863338 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6z6h\" (UniqueName: \"kubernetes.io/projected/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-kube-api-access-v6z6h\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.863565 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.863385 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.863565 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.863422 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.863565 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.863442 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.863565 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.863474 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.863565 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.863497 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.863808 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.863631 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.864265 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.864239 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.865816 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.865790 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.865933 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.865857 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.865933 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.865899 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.865933 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.865901 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.893656 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.893584 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6z6h\" (UniqueName: \"kubernetes.io/projected/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-kube-api-access-v6z6h\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.906050 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.906025 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-v9xq8\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:32.948136 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:32.948110 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:33.257064 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:33.257040 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8"] Oct 30 11:57:33.258621 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:57:33.258592 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4263fb1_dd4f_484d_9b3d_fa3915db3eba.slice/crio-459ef11e287b7bfc378e233738480869986b27bfb812f8a9c668b7541388f44e WatchSource:0}: Error finding container 459ef11e287b7bfc378e233738480869986b27bfb812f8a9c668b7541388f44e: Status 404 returned error can't find the container with id 459ef11e287b7bfc378e233738480869986b27bfb812f8a9c668b7541388f44e Oct 30 11:57:33.863741 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:33.863705 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" event={"ID":"f4263fb1-dd4f-484d-9b3d-fa3915db3eba","Type":"ContainerStarted","Data":"459ef11e287b7bfc378e233738480869986b27bfb812f8a9c668b7541388f44e"} Oct 30 11:57:35.765093 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:35.765051 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:57:35.765387 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:35.765127 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:57:35.871878 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:35.871843 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" event={"ID":"f4263fb1-dd4f-484d-9b3d-fa3915db3eba","Type":"ContainerStarted","Data":"f093d7e94e90b5c3c264fd82924cb279224ed7fb8d57ac0327bf88c6f0086d8f"} Oct 30 11:57:35.872089 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:35.871937 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:35.939714 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:35.939662 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" podStartSLOduration=1.435839912 podStartE2EDuration="3.939648921s" totalImagesPullingTime="2.503809009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:57:35.928209281 +0000 UTC m=+206.962347678" watchObservedRunningTime="2026-10-30 11:57:35.939648921 +0000 UTC m=+206.973787317" Oct 30 11:57:36.877190 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:36.877146 4468 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-v9xq8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 11:57:36.877590 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:36.877216 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" podUID="f4263fb1-dd4f-484d-9b3d-fa3915db3eba" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:57:39.876263 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:57:39.876225 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:57:53.597408 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:57:53.597333 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 11:58:19.864632 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:19.864595 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7"] Oct 30 11:58:19.867585 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:19.867570 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:19.898277 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:19.898248 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7"] Oct 30 11:58:19.913448 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:19.913422 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b713951-98f6-43c2-8e7b-922286766f57-metrics-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-wztn7\" (UID: \"0b713951-98f6-43c2-8e7b-922286766f57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:19.913562 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:19.913469 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0b713951-98f6-43c2-8e7b-922286766f57-manifests\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-wztn7\" (UID: \"0b713951-98f6-43c2-8e7b-922286766f57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:19.913562 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:19.913486 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv5b9\" (UniqueName: \"kubernetes.io/projected/0b713951-98f6-43c2-8e7b-922286766f57-kube-api-access-hv5b9\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-wztn7\" (UID: \"0b713951-98f6-43c2-8e7b-922286766f57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:19.913632 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:19.913569 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0b713951-98f6-43c2-8e7b-922286766f57-webhook-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-wztn7\" (UID: \"0b713951-98f6-43c2-8e7b-922286766f57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:19.913632 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:19.913613 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0b713951-98f6-43c2-8e7b-922286766f57-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-wztn7\" (UID: \"0b713951-98f6-43c2-8e7b-922286766f57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:19.913697 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:19.913647 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b713951-98f6-43c2-8e7b-922286766f57-tmp\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-wztn7\" (UID: \"0b713951-98f6-43c2-8e7b-922286766f57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:19.947131 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:19.947090 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 11:58:19.947131 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:19.947110 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 11:58:19.947131 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:19.947118 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 11:58:19.947331 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:19.947139 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-b7n4r\"" Oct 30 11:58:19.950527 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:19.950510 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 11:58:19.950728 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:19.950715 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 11:58:20.014383 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:20.014350 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0b713951-98f6-43c2-8e7b-922286766f57-manifests\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-wztn7\" (UID: \"0b713951-98f6-43c2-8e7b-922286766f57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:20.014534 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:20.014387 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv5b9\" (UniqueName: \"kubernetes.io/projected/0b713951-98f6-43c2-8e7b-922286766f57-kube-api-access-hv5b9\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-wztn7\" (UID: \"0b713951-98f6-43c2-8e7b-922286766f57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:20.014534 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:20.014421 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0b713951-98f6-43c2-8e7b-922286766f57-webhook-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-wztn7\" (UID: \"0b713951-98f6-43c2-8e7b-922286766f57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:20.014534 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:20.014455 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0b713951-98f6-43c2-8e7b-922286766f57-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-wztn7\" (UID: \"0b713951-98f6-43c2-8e7b-922286766f57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:20.014534 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:20.014501 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b713951-98f6-43c2-8e7b-922286766f57-tmp\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-wztn7\" (UID: \"0b713951-98f6-43c2-8e7b-922286766f57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:20.014732 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:20.014550 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b713951-98f6-43c2-8e7b-922286766f57-metrics-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-wztn7\" (UID: \"0b713951-98f6-43c2-8e7b-922286766f57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:20.014820 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:20.014797 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0b713951-98f6-43c2-8e7b-922286766f57-manifests\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-wztn7\" (UID: \"0b713951-98f6-43c2-8e7b-922286766f57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:20.016737 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:20.016710 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b713951-98f6-43c2-8e7b-922286766f57-tmp\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-wztn7\" (UID: \"0b713951-98f6-43c2-8e7b-922286766f57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:20.017007 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:20.016988 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0b713951-98f6-43c2-8e7b-922286766f57-webhook-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-wztn7\" (UID: \"0b713951-98f6-43c2-8e7b-922286766f57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:20.017117 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:20.017021 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b713951-98f6-43c2-8e7b-922286766f57-metrics-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-wztn7\" (UID: \"0b713951-98f6-43c2-8e7b-922286766f57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:20.017184 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:20.017158 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0b713951-98f6-43c2-8e7b-922286766f57-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-wztn7\" (UID: \"0b713951-98f6-43c2-8e7b-922286766f57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:20.044524 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:20.044501 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv5b9\" (UniqueName: \"kubernetes.io/projected/0b713951-98f6-43c2-8e7b-922286766f57-kube-api-access-hv5b9\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-wztn7\" (UID: \"0b713951-98f6-43c2-8e7b-922286766f57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:20.181133 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:20.181108 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:20.549729 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:58:20.549701 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b713951_98f6_43c2_8e7b_922286766f57.slice/crio-722babab8210b1d25d4416e716640f977b027564ca19749f33ca93e0adf13238 WatchSource:0}: Error finding container 722babab8210b1d25d4416e716640f977b027564ca19749f33ca93e0adf13238: Status 404 returned error can't find the container with id 722babab8210b1d25d4416e716640f977b027564ca19749f33ca93e0adf13238 Oct 30 11:58:20.559083 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:20.559047 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7"] Oct 30 11:58:21.027377 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:21.027345 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" event={"ID":"0b713951-98f6-43c2-8e7b-922286766f57","Type":"ContainerStarted","Data":"722babab8210b1d25d4416e716640f977b027564ca19749f33ca93e0adf13238"} Oct 30 11:58:23.596589 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:58:23.596553 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 11:58:24.040118 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:24.040062 4468 generic.go:356] "Generic (PLEG): container finished" podID="0b713951-98f6-43c2-8e7b-922286766f57" containerID="4b8e783a6ff98eeb222e9beb9d44110b762399032738286ee87734345ff37686" exitCode=0 Oct 30 11:58:24.040298 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:24.040165 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" event={"ID":"0b713951-98f6-43c2-8e7b-922286766f57","Type":"ContainerDied","Data":"4b8e783a6ff98eeb222e9beb9d44110b762399032738286ee87734345ff37686"} Oct 30 11:58:25.048982 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:25.048946 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" event={"ID":"0b713951-98f6-43c2-8e7b-922286766f57","Type":"ContainerStarted","Data":"74fe905771467004c1f3dadd24bc5765ee504ddac517e871d75404e6225a3216"} Oct 30 11:58:25.049667 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:25.049059 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:25.146120 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:25.146041 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" podStartSLOduration=2.5538720230000003 podStartE2EDuration="6.146023547s" totalImagesPullingTime="3.104532338s" totalInitContainerRuntime="487.619186ms" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:25.145545265 +0000 UTC m=+256.179683662" watchObservedRunningTime="2026-10-30 11:58:25.146023547 +0000 UTC m=+256.180161944" Oct 30 11:58:36.054641 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:36.054607 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-wztn7" Oct 30 11:58:51.314873 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:51.314835 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68fcc45f6c-fbxv7"] Oct 30 11:58:51.318157 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:51.318138 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-fbxv7" Oct 30 11:58:51.339202 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:51.339165 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68fcc45f6c-fbxv7"] Oct 30 11:58:51.349199 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:51.349168 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 11:58:51.349199 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:51.349190 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 11:58:51.349381 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:51.349223 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-n7s7n\"" Oct 30 11:58:51.479042 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:51.479002 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4d73c96-8ad8-4aba-908c-88f4969524c1-tls-certs\") pod \"kube-auth-proxy-68fcc45f6c-fbxv7\" (UID: \"b4d73c96-8ad8-4aba-908c-88f4969524c1\") " pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-fbxv7" Oct 30 11:58:51.479298 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:51.479056 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4d73c96-8ad8-4aba-908c-88f4969524c1-tmp\") pod \"kube-auth-proxy-68fcc45f6c-fbxv7\" (UID: \"b4d73c96-8ad8-4aba-908c-88f4969524c1\") " pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-fbxv7" Oct 30 11:58:51.479298 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:51.479109 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrxmg\" (UniqueName: \"kubernetes.io/projected/b4d73c96-8ad8-4aba-908c-88f4969524c1-kube-api-access-hrxmg\") pod \"kube-auth-proxy-68fcc45f6c-fbxv7\" (UID: \"b4d73c96-8ad8-4aba-908c-88f4969524c1\") " pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-fbxv7" Oct 30 11:58:51.579691 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:51.579613 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4d73c96-8ad8-4aba-908c-88f4969524c1-tls-certs\") pod \"kube-auth-proxy-68fcc45f6c-fbxv7\" (UID: \"b4d73c96-8ad8-4aba-908c-88f4969524c1\") " pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-fbxv7" Oct 30 11:58:51.579691 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:51.579648 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4d73c96-8ad8-4aba-908c-88f4969524c1-tmp\") pod \"kube-auth-proxy-68fcc45f6c-fbxv7\" (UID: \"b4d73c96-8ad8-4aba-908c-88f4969524c1\") " pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-fbxv7" Oct 30 11:58:51.579691 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:51.579674 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrxmg\" (UniqueName: \"kubernetes.io/projected/b4d73c96-8ad8-4aba-908c-88f4969524c1-kube-api-access-hrxmg\") pod \"kube-auth-proxy-68fcc45f6c-fbxv7\" (UID: \"b4d73c96-8ad8-4aba-908c-88f4969524c1\") " pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-fbxv7" Oct 30 11:58:51.581955 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:51.581931 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4d73c96-8ad8-4aba-908c-88f4969524c1-tmp\") pod \"kube-auth-proxy-68fcc45f6c-fbxv7\" (UID: \"b4d73c96-8ad8-4aba-908c-88f4969524c1\") " pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-fbxv7" Oct 30 11:58:51.582289 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:51.582267 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4d73c96-8ad8-4aba-908c-88f4969524c1-tls-certs\") pod \"kube-auth-proxy-68fcc45f6c-fbxv7\" (UID: \"b4d73c96-8ad8-4aba-908c-88f4969524c1\") " pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-fbxv7" Oct 30 11:58:51.608254 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:51.608226 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrxmg\" (UniqueName: \"kubernetes.io/projected/b4d73c96-8ad8-4aba-908c-88f4969524c1-kube-api-access-hrxmg\") pod \"kube-auth-proxy-68fcc45f6c-fbxv7\" (UID: \"b4d73c96-8ad8-4aba-908c-88f4969524c1\") " pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-fbxv7" Oct 30 11:58:51.629933 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:51.629902 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-fbxv7" Oct 30 11:58:51.934001 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:51.933968 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68fcc45f6c-fbxv7"] Oct 30 11:58:51.936226 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:58:51.936196 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d73c96_8ad8_4aba_908c_88f4969524c1.slice/crio-9c95462dea1995eb770cf2d8f3e17e4dbe9560c80066bbb1ccad788da69065d8 WatchSource:0}: Error finding container 9c95462dea1995eb770cf2d8f3e17e4dbe9560c80066bbb1ccad788da69065d8: Status 404 returned error can't find the container with id 9c95462dea1995eb770cf2d8f3e17e4dbe9560c80066bbb1ccad788da69065d8 Oct 30 11:58:52.145262 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:52.145228 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-fbxv7" event={"ID":"b4d73c96-8ad8-4aba-908c-88f4969524c1","Type":"ContainerStarted","Data":"9c95462dea1995eb770cf2d8f3e17e4dbe9560c80066bbb1ccad788da69065d8"} Oct 30 11:58:53.599421 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:58:53.599380 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 11:58:55.157886 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:55.157849 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-fbxv7" event={"ID":"b4d73c96-8ad8-4aba-908c-88f4969524c1","Type":"ContainerStarted","Data":"9c830a7b5055391f39c1ea21eaaefefd1b558ef06dfe608ae0c7fab59240a473"} Oct 30 11:58:55.197118 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:58:55.197038 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-fbxv7" podStartSLOduration=1.895180498 podStartE2EDuration="4.197022253s" totalImagesPullingTime="2.301841755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:55.196620067 +0000 UTC m=+286.230758462" watchObservedRunningTime="2026-10-30 11:58:55.197022253 +0000 UTC m=+286.231160642" Oct 30 11:59:23.597121 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:59:23.597005 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 11:59:53.599628 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:59:53.599590 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 11:59:54.071896 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.071862 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv"] Oct 30 11:59:54.074957 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.074934 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.103743 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.103713 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv"] Oct 30 11:59:54.176329 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.176290 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.176532 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.176405 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.176613 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.176538 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rwvd\" (UniqueName: \"kubernetes.io/projected/5929f380-6cc0-4ccf-97b8-a3e54b01430f-kube-api-access-7rwvd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.176712 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.176695 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.176805 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.176789 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.176854 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.176825 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.176916 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.176889 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5929f380-6cc0-4ccf-97b8-a3e54b01430f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.277478 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.277435 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.277666 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.277490 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.277666 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.277517 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.277666 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.277545 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5929f380-6cc0-4ccf-97b8-a3e54b01430f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.277666 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.277578 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.277666 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.277615 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.277910 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.277731 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rwvd\" (UniqueName: \"kubernetes.io/projected/5929f380-6cc0-4ccf-97b8-a3e54b01430f-kube-api-access-7rwvd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.278567 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.278531 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.280974 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.280934 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5929f380-6cc0-4ccf-97b8-a3e54b01430f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.281151 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.281131 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.281151 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.281140 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.281272 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.281207 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.300735 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.300697 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.300920 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.300894 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rwvd\" (UniqueName: \"kubernetes.io/projected/5929f380-6cc0-4ccf-97b8-a3e54b01430f-kube-api-access-7rwvd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h89vv\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.388827 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.388736 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:54.716452 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.716425 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv"] Oct 30 11:59:54.717996 ip-10-0-131-76 kubenswrapper[4468]: W1030 11:59:54.717972 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5929f380_6cc0_4ccf_97b8_a3e54b01430f.slice/crio-1ecd97670f57c65bdc2c05c7aa753235466d63f3366a5ca36e68caa0bb0b3ebc WatchSource:0}: Error finding container 1ecd97670f57c65bdc2c05c7aa753235466d63f3366a5ca36e68caa0bb0b3ebc: Status 404 returned error can't find the container with id 1ecd97670f57c65bdc2c05c7aa753235466d63f3366a5ca36e68caa0bb0b3ebc Oct 30 11:59:54.720180 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.720148 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:59:54.720275 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:54.720212 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:59:55.378923 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:55.378886 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" event={"ID":"5929f380-6cc0-4ccf-97b8-a3e54b01430f","Type":"ContainerStarted","Data":"7c8414bc728f0a4b4b4aaf40400c1f9d5bc1d1fbc11362cff6d2f3f6e2e83abe"} Oct 30 11:59:55.379154 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:55.378937 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" event={"ID":"5929f380-6cc0-4ccf-97b8-a3e54b01430f","Type":"ContainerStarted","Data":"1ecd97670f57c65bdc2c05c7aa753235466d63f3366a5ca36e68caa0bb0b3ebc"} Oct 30 11:59:55.379239 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:55.379169 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:55.380571 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:55.380547 4468 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-h89vv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 11:59:55.380686 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:55.380604 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" podUID="5929f380-6cc0-4ccf-97b8-a3e54b01430f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:59:55.430312 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:55.430254 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" podStartSLOduration=1.430236794 podStartE2EDuration="1.430236794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:55.429093012 +0000 UTC m=+346.463231402" watchObservedRunningTime="2026-10-30 11:59:55.430236794 +0000 UTC m=+346.464375190" Oct 30 11:59:56.382982 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:56.382956 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 11:59:56.473941 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:56.473907 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8"] Oct 30 11:59:56.474156 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:56.474125 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" podUID="f4263fb1-dd4f-484d-9b3d-fa3915db3eba" containerName="discovery" containerID="cri-o://f093d7e94e90b5c3c264fd82924cb279224ed7fb8d57ac0327bf88c6f0086d8f" gracePeriod=30 Oct 30 11:59:56.718176 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:56.718144 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:59:56.904751 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:56.904665 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-local-certs\") pod \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " Oct 30 11:59:56.904751 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:56.904714 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-csr-ca-configmap\") pod \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " Oct 30 11:59:56.904994 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:56.904768 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-cacerts\" (UniqueName: \"kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-cacerts\") pod \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " Oct 30 11:59:56.904994 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:56.904794 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-csr-dns-cert\") pod \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " Oct 30 11:59:56.904994 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:56.904842 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-token\" (UniqueName: \"kubernetes.io/projected/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-token\") pod \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " Oct 30 11:59:56.904994 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:56.904976 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-kubeconfig\") pod \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " Oct 30 11:59:56.905237 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:56.905027 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-kube-api-access-v6z6h\" (UniqueName: \"kubernetes.io/projected/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-kube-api-access-v6z6h\") pod \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\" (UID: \"f4263fb1-dd4f-484d-9b3d-fa3915db3eba\") " Oct 30 11:59:56.905289 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:56.905231 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-csr-ca-configmap" pod "f4263fb1-dd4f-484d-9b3d-fa3915db3eba" (UID: "f4263fb1-dd4f-484d-9b3d-fa3915db3eba"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:59:56.905350 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:56.905331 4468 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-csr-ca-configmap\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:59:56.907168 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:56.907140 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-csr-dns-cert" pod "f4263fb1-dd4f-484d-9b3d-fa3915db3eba" (UID: "f4263fb1-dd4f-484d-9b3d-fa3915db3eba"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:59:56.907307 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:56.907228 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-local-certs" pod "f4263fb1-dd4f-484d-9b3d-fa3915db3eba" (UID: "f4263fb1-dd4f-484d-9b3d-fa3915db3eba"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:59:56.907478 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:56.907456 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-kubeconfig" pod "f4263fb1-dd4f-484d-9b3d-fa3915db3eba" (UID: "f4263fb1-dd4f-484d-9b3d-fa3915db3eba"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:59:56.907566 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:56.907473 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-cacerts" pod "f4263fb1-dd4f-484d-9b3d-fa3915db3eba" (UID: "f4263fb1-dd4f-484d-9b3d-fa3915db3eba"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:59:56.907638 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:56.907615 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-kube-api-access-v6z6h" pod "f4263fb1-dd4f-484d-9b3d-fa3915db3eba" (UID: "f4263fb1-dd4f-484d-9b3d-fa3915db3eba"). InnerVolumeSpecName "kube-api-access-v6z6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:59:56.908454 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:56.908427 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-token" pod "f4263fb1-dd4f-484d-9b3d-fa3915db3eba" (UID: "f4263fb1-dd4f-484d-9b3d-fa3915db3eba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:59:57.006699 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:57.006667 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6z6h\" (UniqueName: \"kubernetes.io/projected/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-kube-api-access-v6z6h\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:59:57.006699 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:57.006697 4468 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-local-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:59:57.006699 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:57.006707 4468 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-cacerts\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:59:57.006699 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:57.006715 4468 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-csr-dns-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:59:57.006961 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:57.006724 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:59:57.006961 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:57.006733 4468 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f4263fb1-dd4f-484d-9b3d-fa3915db3eba-istio-kubeconfig\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 11:59:57.387357 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:57.387319 4468 generic.go:356] "Generic (PLEG): container finished" podID="f4263fb1-dd4f-484d-9b3d-fa3915db3eba" containerID="f093d7e94e90b5c3c264fd82924cb279224ed7fb8d57ac0327bf88c6f0086d8f" exitCode=0 Oct 30 11:59:57.387811 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:57.387373 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" Oct 30 11:59:57.387811 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:57.387401 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" event={"ID":"f4263fb1-dd4f-484d-9b3d-fa3915db3eba","Type":"ContainerDied","Data":"f093d7e94e90b5c3c264fd82924cb279224ed7fb8d57ac0327bf88c6f0086d8f"} Oct 30 11:59:57.387811 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:57.387451 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8" event={"ID":"f4263fb1-dd4f-484d-9b3d-fa3915db3eba","Type":"ContainerDied","Data":"459ef11e287b7bfc378e233738480869986b27bfb812f8a9c668b7541388f44e"} Oct 30 11:59:57.387811 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:57.387472 4468 scope.go:122] "RemoveContainer" containerID="f093d7e94e90b5c3c264fd82924cb279224ed7fb8d57ac0327bf88c6f0086d8f" Oct 30 11:59:57.395625 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:57.395601 4468 scope.go:122] "RemoveContainer" containerID="f093d7e94e90b5c3c264fd82924cb279224ed7fb8d57ac0327bf88c6f0086d8f" Oct 30 11:59:57.395926 ip-10-0-131-76 kubenswrapper[4468]: E1030 11:59:57.395908 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f093d7e94e90b5c3c264fd82924cb279224ed7fb8d57ac0327bf88c6f0086d8f\": container with ID starting with f093d7e94e90b5c3c264fd82924cb279224ed7fb8d57ac0327bf88c6f0086d8f not found: ID does not exist" containerID="f093d7e94e90b5c3c264fd82924cb279224ed7fb8d57ac0327bf88c6f0086d8f" Oct 30 11:59:57.395971 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:57.395937 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f093d7e94e90b5c3c264fd82924cb279224ed7fb8d57ac0327bf88c6f0086d8f"} err="failed to get container status \"f093d7e94e90b5c3c264fd82924cb279224ed7fb8d57ac0327bf88c6f0086d8f\": rpc error: code = NotFound desc = could not find container \"f093d7e94e90b5c3c264fd82924cb279224ed7fb8d57ac0327bf88c6f0086d8f\": container with ID starting with f093d7e94e90b5c3c264fd82924cb279224ed7fb8d57ac0327bf88c6f0086d8f not found: ID does not exist" Oct 30 11:59:57.439475 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:57.439444 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8"] Oct 30 11:59:57.451252 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:57.451222 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-v9xq8"] Oct 30 11:59:59.087737 ip-10-0-131-76 kubenswrapper[4468]: I1030 11:59:59.087700 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4263fb1-dd4f-484d-9b3d-fa3915db3eba" path="/var/lib/kubelet/pods/f4263fb1-dd4f-484d-9b3d-fa3915db3eba/volumes" Oct 30 12:00:23.597397 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:00:23.597351 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:00:53.597823 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:00:53.597749 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:01:23.597013 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:01:23.596952 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:01:53.596452 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:01:53.596409 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:02:23.597026 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:02:23.596943 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:02:53.597081 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:02:53.597035 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:03:23.596765 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:03:23.596724 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:03:53.597350 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:03:53.597269 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:04:23.597305 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:04:23.597268 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:04:53.596496 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:04:53.596455 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:05:23.597804 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:05:23.597725 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:05:53.595787 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:05:53.595743 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:06:23.598396 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:06:23.598349 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:06:53.596626 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:06:53.596552 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:07:23.597202 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:07:23.597168 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:07:53.597267 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:07:53.597227 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:08:23.597115 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:08:23.597021 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:08:53.596560 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:08:53.596520 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:09:23.596950 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:09:23.596912 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:09:31.013534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.013497 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wmzfj"] Oct 30 12:09:31.014156 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.013912 4468 cpu_manager.go:414] "containerMap: removing container" podUID="f4263fb1-dd4f-484d-9b3d-fa3915db3eba" containerName="discovery" Oct 30 12:09:31.014156 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.013928 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4263fb1-dd4f-484d-9b3d-fa3915db3eba" containerName="discovery" Oct 30 12:09:31.014156 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.014029 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4263fb1-dd4f-484d-9b3d-fa3915db3eba" containerName="discovery" Oct 30 12:09:31.016810 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.016793 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wmzfj" Oct 30 12:09:31.033948 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.033915 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 12:09:31.033948 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.033936 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 12:09:31.042616 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.042579 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 12:09:31.042616 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.042622 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jg5cf\"" Oct 30 12:09:31.043917 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.043887 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wmzfj"] Oct 30 12:09:31.127130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.127094 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a5752a9c-ee5c-43df-8db6-e346f3f3c387-config-file\") pod \"limitador-limitador-74dfcfb894-wmzfj\" (UID: \"a5752a9c-ee5c-43df-8db6-e346f3f3c387\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wmzfj" Oct 30 12:09:31.127286 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.127257 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x4b6\" (UniqueName: \"kubernetes.io/projected/a5752a9c-ee5c-43df-8db6-e346f3f3c387-kube-api-access-7x4b6\") pod \"limitador-limitador-74dfcfb894-wmzfj\" (UID: \"a5752a9c-ee5c-43df-8db6-e346f3f3c387\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wmzfj" Oct 30 12:09:31.153095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.153018 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wmzfj"] Oct 30 12:09:31.228231 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.228196 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x4b6\" (UniqueName: \"kubernetes.io/projected/a5752a9c-ee5c-43df-8db6-e346f3f3c387-kube-api-access-7x4b6\") pod \"limitador-limitador-74dfcfb894-wmzfj\" (UID: \"a5752a9c-ee5c-43df-8db6-e346f3f3c387\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wmzfj" Oct 30 12:09:31.228471 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.228257 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a5752a9c-ee5c-43df-8db6-e346f3f3c387-config-file\") pod \"limitador-limitador-74dfcfb894-wmzfj\" (UID: \"a5752a9c-ee5c-43df-8db6-e346f3f3c387\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wmzfj" Oct 30 12:09:31.229272 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.229244 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a5752a9c-ee5c-43df-8db6-e346f3f3c387-config-file\") pod \"limitador-limitador-74dfcfb894-wmzfj\" (UID: \"a5752a9c-ee5c-43df-8db6-e346f3f3c387\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wmzfj" Oct 30 12:09:31.250708 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.250667 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x4b6\" (UniqueName: \"kubernetes.io/projected/a5752a9c-ee5c-43df-8db6-e346f3f3c387-kube-api-access-7x4b6\") pod \"limitador-limitador-74dfcfb894-wmzfj\" (UID: \"a5752a9c-ee5c-43df-8db6-e346f3f3c387\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wmzfj" Oct 30 12:09:31.329362 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.329266 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wmzfj" Oct 30 12:09:31.627446 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.627423 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wmzfj"] Oct 30 12:09:31.629307 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:09:31.629266 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5752a9c_ee5c_43df_8db6_e346f3f3c387.slice/crio-65e498b233f24b8383425227e0fe8e45f99ecc299cf0111382f4217711415541 WatchSource:0}: Error finding container 65e498b233f24b8383425227e0fe8e45f99ecc299cf0111382f4217711415541: Status 404 returned error can't find the container with id 65e498b233f24b8383425227e0fe8e45f99ecc299cf0111382f4217711415541 Oct 30 12:09:31.631429 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.631410 4468 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:09:31.874787 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.874750 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-d56wv"] Oct 30 12:09:31.879211 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.879153 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-d56wv" Oct 30 12:09:31.893840 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.893818 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bs7kx\"" Oct 30 12:09:31.895234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:31.895217 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-d56wv"] Oct 30 12:09:32.034128 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:32.034065 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbgnb\" (UniqueName: \"kubernetes.io/projected/24ed85f4-bfd7-444f-a280-ca82986730ec-kube-api-access-gbgnb\") pod \"authorino-57f7f8659f-d56wv\" (UID: \"24ed85f4-bfd7-444f-a280-ca82986730ec\") " pod="kuadrant-system/authorino-57f7f8659f-d56wv" Oct 30 12:09:32.134645 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:32.134556 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbgnb\" (UniqueName: \"kubernetes.io/projected/24ed85f4-bfd7-444f-a280-ca82986730ec-kube-api-access-gbgnb\") pod \"authorino-57f7f8659f-d56wv\" (UID: \"24ed85f4-bfd7-444f-a280-ca82986730ec\") " pod="kuadrant-system/authorino-57f7f8659f-d56wv" Oct 30 12:09:32.160437 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:32.160407 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbgnb\" (UniqueName: \"kubernetes.io/projected/24ed85f4-bfd7-444f-a280-ca82986730ec-kube-api-access-gbgnb\") pod \"authorino-57f7f8659f-d56wv\" (UID: \"24ed85f4-bfd7-444f-a280-ca82986730ec\") " pod="kuadrant-system/authorino-57f7f8659f-d56wv" Oct 30 12:09:32.208129 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:32.208102 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-d56wv" Oct 30 12:09:32.482368 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:32.482332 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-vbmqz"] Oct 30 12:09:32.485828 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:32.485799 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-vbmqz" Oct 30 12:09:32.505272 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:32.505249 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wmzfj" event={"ID":"a5752a9c-ee5c-43df-8db6-e346f3f3c387","Type":"ContainerStarted","Data":"65e498b233f24b8383425227e0fe8e45f99ecc299cf0111382f4217711415541"} Oct 30 12:09:32.505389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:32.505301 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-vbmqz"] Oct 30 12:09:32.509778 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:32.509729 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-d56wv"] Oct 30 12:09:32.512648 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:09:32.512624 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24ed85f4_bfd7_444f_a280_ca82986730ec.slice/crio-52be768bbe2929459c8f0714ecbd82bb60eee5a699ef24e6edfe2f595496109a WatchSource:0}: Error finding container 52be768bbe2929459c8f0714ecbd82bb60eee5a699ef24e6edfe2f595496109a: Status 404 returned error can't find the container with id 52be768bbe2929459c8f0714ecbd82bb60eee5a699ef24e6edfe2f595496109a Oct 30 12:09:32.638791 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:32.638757 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kxxp\" (UniqueName: \"kubernetes.io/projected/38e3670c-3e3a-4e25-a591-a5804c478e5b-kube-api-access-4kxxp\") pod \"authorino-56c6c9ddd5-vbmqz\" (UID: \"38e3670c-3e3a-4e25-a591-a5804c478e5b\") " pod="kuadrant-system/authorino-56c6c9ddd5-vbmqz" Oct 30 12:09:32.739854 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:32.739770 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kxxp\" (UniqueName: \"kubernetes.io/projected/38e3670c-3e3a-4e25-a591-a5804c478e5b-kube-api-access-4kxxp\") pod \"authorino-56c6c9ddd5-vbmqz\" (UID: \"38e3670c-3e3a-4e25-a591-a5804c478e5b\") " pod="kuadrant-system/authorino-56c6c9ddd5-vbmqz" Oct 30 12:09:32.764821 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:32.764780 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kxxp\" (UniqueName: \"kubernetes.io/projected/38e3670c-3e3a-4e25-a591-a5804c478e5b-kube-api-access-4kxxp\") pod \"authorino-56c6c9ddd5-vbmqz\" (UID: \"38e3670c-3e3a-4e25-a591-a5804c478e5b\") " pod="kuadrant-system/authorino-56c6c9ddd5-vbmqz" Oct 30 12:09:32.799637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:32.799605 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-vbmqz" Oct 30 12:09:33.104937 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:33.104912 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-vbmqz"] Oct 30 12:09:33.106283 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:09:33.106259 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38e3670c_3e3a_4e25_a591_a5804c478e5b.slice/crio-b60a5cb970af8cf838c4048aa2f1e46d981d3b02c5fd62ae34a21af46bcc2000 WatchSource:0}: Error finding container b60a5cb970af8cf838c4048aa2f1e46d981d3b02c5fd62ae34a21af46bcc2000: Status 404 returned error can't find the container with id b60a5cb970af8cf838c4048aa2f1e46d981d3b02c5fd62ae34a21af46bcc2000 Oct 30 12:09:33.512908 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:33.512823 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-d56wv" event={"ID":"24ed85f4-bfd7-444f-a280-ca82986730ec","Type":"ContainerStarted","Data":"52be768bbe2929459c8f0714ecbd82bb60eee5a699ef24e6edfe2f595496109a"} Oct 30 12:09:33.514502 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:33.514442 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-vbmqz" event={"ID":"38e3670c-3e3a-4e25-a591-a5804c478e5b","Type":"ContainerStarted","Data":"b60a5cb970af8cf838c4048aa2f1e46d981d3b02c5fd62ae34a21af46bcc2000"} Oct 30 12:09:34.520850 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:34.520814 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wmzfj" event={"ID":"a5752a9c-ee5c-43df-8db6-e346f3f3c387","Type":"ContainerStarted","Data":"6ef1d63d2fc5b94071f7909f14a8ff3ab72ba2c74c9a0ccf572333a7cab7afd7"} Oct 30 12:09:34.521318 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:34.521103 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-wmzfj" Oct 30 12:09:34.561699 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:34.561586 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-wmzfj" podStartSLOduration=1.8927773509999999 podStartE2EDuration="4.561570412s" totalImagesPullingTime="2.668793061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:09:34.560423345 +0000 UTC m=+925.594561741" watchObservedRunningTime="2026-10-30 12:09:34.561570412 +0000 UTC m=+925.595708808" Oct 30 12:09:35.526148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:35.526103 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-d56wv" event={"ID":"24ed85f4-bfd7-444f-a280-ca82986730ec","Type":"ContainerStarted","Data":"ac156c80b89a6b4b5493e3d57fd91ab439e6640b4d4c8e379439a45772caad54"} Oct 30 12:09:35.527677 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:35.527647 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-vbmqz" event={"ID":"38e3670c-3e3a-4e25-a591-a5804c478e5b","Type":"ContainerStarted","Data":"8d1b0d352a3b2cb73f9895c02510f8c208d08ad4310a2f7bb76aba8ff121d51a"} Oct 30 12:09:35.563992 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:35.563934 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-d56wv" podStartSLOduration=2.225365843 podStartE2EDuration="4.563919362s" totalImagesPullingTime="2.338553519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:09:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:09:35.563323317 +0000 UTC m=+926.597461713" watchObservedRunningTime="2026-10-30 12:09:35.563919362 +0000 UTC m=+926.598057757" Oct 30 12:09:35.602128 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:35.602043 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-vbmqz" podStartSLOduration=1.649865579 podStartE2EDuration="3.60202358s" totalImagesPullingTime="1.952158001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:09:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:09:35.600944847 +0000 UTC m=+926.635083244" watchObservedRunningTime="2026-10-30 12:09:35.60202358 +0000 UTC m=+926.636161978" Oct 30 12:09:35.624585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:35.624552 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-d56wv"] Oct 30 12:09:37.537771 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:37.537732 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-d56wv" podUID="24ed85f4-bfd7-444f-a280-ca82986730ec" containerName="authorino" containerID="cri-o://ac156c80b89a6b4b5493e3d57fd91ab439e6640b4d4c8e379439a45772caad54" gracePeriod=30 Oct 30 12:09:37.782181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:37.782155 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-d56wv" Oct 30 12:09:37.881293 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:37.881203 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ed85f4-bfd7-444f-a280-ca82986730ec-kube-api-access-gbgnb\" (UniqueName: \"kubernetes.io/projected/24ed85f4-bfd7-444f-a280-ca82986730ec-kube-api-access-gbgnb\") pod \"24ed85f4-bfd7-444f-a280-ca82986730ec\" (UID: \"24ed85f4-bfd7-444f-a280-ca82986730ec\") " Oct 30 12:09:37.883412 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:37.883388 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ed85f4-bfd7-444f-a280-ca82986730ec-kube-api-access-gbgnb" pod "24ed85f4-bfd7-444f-a280-ca82986730ec" (UID: "24ed85f4-bfd7-444f-a280-ca82986730ec"). InnerVolumeSpecName "kube-api-access-gbgnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:09:37.982552 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:37.982510 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbgnb\" (UniqueName: \"kubernetes.io/projected/24ed85f4-bfd7-444f-a280-ca82986730ec-kube-api-access-gbgnb\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:09:38.542946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:38.542908 4468 generic.go:356] "Generic (PLEG): container finished" podID="24ed85f4-bfd7-444f-a280-ca82986730ec" containerID="ac156c80b89a6b4b5493e3d57fd91ab439e6640b4d4c8e379439a45772caad54" exitCode=0 Oct 30 12:09:38.543448 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:38.542953 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-d56wv" Oct 30 12:09:38.543448 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:38.542993 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-d56wv" event={"ID":"24ed85f4-bfd7-444f-a280-ca82986730ec","Type":"ContainerDied","Data":"ac156c80b89a6b4b5493e3d57fd91ab439e6640b4d4c8e379439a45772caad54"} Oct 30 12:09:38.543448 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:38.543038 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-d56wv" event={"ID":"24ed85f4-bfd7-444f-a280-ca82986730ec","Type":"ContainerDied","Data":"52be768bbe2929459c8f0714ecbd82bb60eee5a699ef24e6edfe2f595496109a"} Oct 30 12:09:38.543448 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:38.543055 4468 scope.go:122] "RemoveContainer" containerID="ac156c80b89a6b4b5493e3d57fd91ab439e6640b4d4c8e379439a45772caad54" Oct 30 12:09:38.551771 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:38.551748 4468 scope.go:122] "RemoveContainer" containerID="ac156c80b89a6b4b5493e3d57fd91ab439e6640b4d4c8e379439a45772caad54" Oct 30 12:09:38.552108 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:09:38.552090 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac156c80b89a6b4b5493e3d57fd91ab439e6640b4d4c8e379439a45772caad54\": container with ID starting with ac156c80b89a6b4b5493e3d57fd91ab439e6640b4d4c8e379439a45772caad54 not found: ID does not exist" containerID="ac156c80b89a6b4b5493e3d57fd91ab439e6640b4d4c8e379439a45772caad54" Oct 30 12:09:38.552208 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:38.552118 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac156c80b89a6b4b5493e3d57fd91ab439e6640b4d4c8e379439a45772caad54"} err="failed to get container status \"ac156c80b89a6b4b5493e3d57fd91ab439e6640b4d4c8e379439a45772caad54\": rpc error: code = NotFound desc = could not find container \"ac156c80b89a6b4b5493e3d57fd91ab439e6640b4d4c8e379439a45772caad54\": container with ID starting with ac156c80b89a6b4b5493e3d57fd91ab439e6640b4d4c8e379439a45772caad54 not found: ID does not exist" Oct 30 12:09:38.591354 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:38.591326 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-d56wv"] Oct 30 12:09:38.602692 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:38.602669 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-d56wv"] Oct 30 12:09:39.088044 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:39.088014 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ed85f4-bfd7-444f-a280-ca82986730ec" path="/var/lib/kubelet/pods/24ed85f4-bfd7-444f-a280-ca82986730ec/volumes" Oct 30 12:09:45.529185 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:45.529156 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-wmzfj" Oct 30 12:09:45.628273 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:45.628236 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wmzfj"] Oct 30 12:09:45.628471 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:45.628428 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-wmzfj" podUID="a5752a9c-ee5c-43df-8db6-e346f3f3c387" containerName="limitador" containerID="cri-o://6ef1d63d2fc5b94071f7909f14a8ff3ab72ba2c74c9a0ccf572333a7cab7afd7" gracePeriod=30 Oct 30 12:09:46.165203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:46.165177 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wmzfj" Oct 30 12:09:46.249131 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:46.249010 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5752a9c-ee5c-43df-8db6-e346f3f3c387-config-file\" (UniqueName: \"kubernetes.io/configmap/a5752a9c-ee5c-43df-8db6-e346f3f3c387-config-file\") pod \"a5752a9c-ee5c-43df-8db6-e346f3f3c387\" (UID: \"a5752a9c-ee5c-43df-8db6-e346f3f3c387\") " Oct 30 12:09:46.249316 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:46.249134 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5752a9c-ee5c-43df-8db6-e346f3f3c387-kube-api-access-7x4b6\" (UniqueName: \"kubernetes.io/projected/a5752a9c-ee5c-43df-8db6-e346f3f3c387-kube-api-access-7x4b6\") pod \"a5752a9c-ee5c-43df-8db6-e346f3f3c387\" (UID: \"a5752a9c-ee5c-43df-8db6-e346f3f3c387\") " Oct 30 12:09:46.249493 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:46.249465 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5752a9c-ee5c-43df-8db6-e346f3f3c387-config-file" pod "a5752a9c-ee5c-43df-8db6-e346f3f3c387" (UID: "a5752a9c-ee5c-43df-8db6-e346f3f3c387"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:09:46.251208 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:46.251186 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5752a9c-ee5c-43df-8db6-e346f3f3c387-kube-api-access-7x4b6" pod "a5752a9c-ee5c-43df-8db6-e346f3f3c387" (UID: "a5752a9c-ee5c-43df-8db6-e346f3f3c387"). InnerVolumeSpecName "kube-api-access-7x4b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:09:46.350382 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:46.350340 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x4b6\" (UniqueName: \"kubernetes.io/projected/a5752a9c-ee5c-43df-8db6-e346f3f3c387-kube-api-access-7x4b6\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:09:46.350382 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:46.350376 4468 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a5752a9c-ee5c-43df-8db6-e346f3f3c387-config-file\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:09:46.576319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:46.576234 4468 generic.go:356] "Generic (PLEG): container finished" podID="a5752a9c-ee5c-43df-8db6-e346f3f3c387" containerID="6ef1d63d2fc5b94071f7909f14a8ff3ab72ba2c74c9a0ccf572333a7cab7afd7" exitCode=0 Oct 30 12:09:46.576319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:46.576283 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wmzfj" Oct 30 12:09:46.576739 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:46.576314 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wmzfj" event={"ID":"a5752a9c-ee5c-43df-8db6-e346f3f3c387","Type":"ContainerDied","Data":"6ef1d63d2fc5b94071f7909f14a8ff3ab72ba2c74c9a0ccf572333a7cab7afd7"} Oct 30 12:09:46.576739 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:46.576368 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wmzfj" event={"ID":"a5752a9c-ee5c-43df-8db6-e346f3f3c387","Type":"ContainerDied","Data":"65e498b233f24b8383425227e0fe8e45f99ecc299cf0111382f4217711415541"} Oct 30 12:09:46.576739 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:46.576390 4468 scope.go:122] "RemoveContainer" containerID="6ef1d63d2fc5b94071f7909f14a8ff3ab72ba2c74c9a0ccf572333a7cab7afd7" Oct 30 12:09:46.591039 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:46.591019 4468 scope.go:122] "RemoveContainer" containerID="6ef1d63d2fc5b94071f7909f14a8ff3ab72ba2c74c9a0ccf572333a7cab7afd7" Oct 30 12:09:46.591350 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:09:46.591331 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef1d63d2fc5b94071f7909f14a8ff3ab72ba2c74c9a0ccf572333a7cab7afd7\": container with ID starting with 6ef1d63d2fc5b94071f7909f14a8ff3ab72ba2c74c9a0ccf572333a7cab7afd7 not found: ID does not exist" containerID="6ef1d63d2fc5b94071f7909f14a8ff3ab72ba2c74c9a0ccf572333a7cab7afd7" Oct 30 12:09:46.591402 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:46.591359 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef1d63d2fc5b94071f7909f14a8ff3ab72ba2c74c9a0ccf572333a7cab7afd7"} err="failed to get container status \"6ef1d63d2fc5b94071f7909f14a8ff3ab72ba2c74c9a0ccf572333a7cab7afd7\": rpc error: code = NotFound desc = could not find container \"6ef1d63d2fc5b94071f7909f14a8ff3ab72ba2c74c9a0ccf572333a7cab7afd7\": container with ID starting with 6ef1d63d2fc5b94071f7909f14a8ff3ab72ba2c74c9a0ccf572333a7cab7afd7 not found: ID does not exist" Oct 30 12:09:46.628823 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:46.628796 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wmzfj"] Oct 30 12:09:46.640393 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:46.640363 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wmzfj"] Oct 30 12:09:47.087478 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:47.087446 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5752a9c-ee5c-43df-8db6-e346f3f3c387" path="/var/lib/kubelet/pods/a5752a9c-ee5c-43df-8db6-e346f3f3c387/volumes" Oct 30 12:09:52.279537 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.279448 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-48ksl"] Oct 30 12:09:52.279918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.279731 4468 cpu_manager.go:414] "containerMap: removing container" podUID="a5752a9c-ee5c-43df-8db6-e346f3f3c387" containerName="limitador" Oct 30 12:09:52.279918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.279741 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5752a9c-ee5c-43df-8db6-e346f3f3c387" containerName="limitador" Oct 30 12:09:52.279918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.279753 4468 cpu_manager.go:414] "containerMap: removing container" podUID="24ed85f4-bfd7-444f-a280-ca82986730ec" containerName="authorino" Oct 30 12:09:52.279918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.279758 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ed85f4-bfd7-444f-a280-ca82986730ec" containerName="authorino" Oct 30 12:09:52.279918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.279813 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ed85f4-bfd7-444f-a280-ca82986730ec" containerName="authorino" Oct 30 12:09:52.279918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.279826 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5752a9c-ee5c-43df-8db6-e346f3f3c387" containerName="limitador" Oct 30 12:09:52.283892 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.283867 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-48ksl" Oct 30 12:09:52.304671 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.304641 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-48ksl"] Oct 30 12:09:52.314808 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.314780 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:09:52.314962 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.314813 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 12:09:52.315152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.315133 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:09:52.315152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.315145 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-qxvtf\"" Oct 30 12:09:52.396473 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.396434 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzhw4\" (UniqueName: \"kubernetes.io/projected/6dd92beb-9078-4354-aa07-01d6ccf44643-kube-api-access-vzhw4\") pod \"postgres-7cdbf6dc44-48ksl\" (UID: \"6dd92beb-9078-4354-aa07-01d6ccf44643\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-48ksl" Oct 30 12:09:52.396627 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.396487 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6dd92beb-9078-4354-aa07-01d6ccf44643-data\") pod \"postgres-7cdbf6dc44-48ksl\" (UID: \"6dd92beb-9078-4354-aa07-01d6ccf44643\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-48ksl" Oct 30 12:09:52.497806 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.497768 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzhw4\" (UniqueName: \"kubernetes.io/projected/6dd92beb-9078-4354-aa07-01d6ccf44643-kube-api-access-vzhw4\") pod \"postgres-7cdbf6dc44-48ksl\" (UID: \"6dd92beb-9078-4354-aa07-01d6ccf44643\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-48ksl" Oct 30 12:09:52.498020 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.497830 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6dd92beb-9078-4354-aa07-01d6ccf44643-data\") pod \"postgres-7cdbf6dc44-48ksl\" (UID: \"6dd92beb-9078-4354-aa07-01d6ccf44643\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-48ksl" Oct 30 12:09:52.498286 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.498265 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6dd92beb-9078-4354-aa07-01d6ccf44643-data\") pod \"postgres-7cdbf6dc44-48ksl\" (UID: \"6dd92beb-9078-4354-aa07-01d6ccf44643\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-48ksl" Oct 30 12:09:52.520572 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.520549 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzhw4\" (UniqueName: \"kubernetes.io/projected/6dd92beb-9078-4354-aa07-01d6ccf44643-kube-api-access-vzhw4\") pod \"postgres-7cdbf6dc44-48ksl\" (UID: \"6dd92beb-9078-4354-aa07-01d6ccf44643\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-48ksl" Oct 30 12:09:52.596439 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.596347 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-48ksl" Oct 30 12:09:52.897245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:52.897216 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-48ksl"] Oct 30 12:09:52.897651 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:09:52.897627 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd92beb_9078_4354_aa07_01d6ccf44643.slice/crio-dcfe34e592dc4d6073b590743b574173ff38960cb3cdde7784272c8e220ce776 WatchSource:0}: Error finding container dcfe34e592dc4d6073b590743b574173ff38960cb3cdde7784272c8e220ce776: Status 404 returned error can't find the container with id dcfe34e592dc4d6073b590743b574173ff38960cb3cdde7784272c8e220ce776 Oct 30 12:09:53.610575 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:09:53.610538 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:09:53.613907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:53.613833 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-48ksl" event={"ID":"6dd92beb-9078-4354-aa07-01d6ccf44643","Type":"ContainerStarted","Data":"dcfe34e592dc4d6073b590743b574173ff38960cb3cdde7784272c8e220ce776"} Oct 30 12:09:58.636691 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:58.636656 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-48ksl" event={"ID":"6dd92beb-9078-4354-aa07-01d6ccf44643","Type":"ContainerStarted","Data":"cb73b3ec1e9b03f51e52ba4e5699234d4ea2a7d431a0daee8d55ce9415a41abf"} Oct 30 12:09:58.637207 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:58.636831 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-48ksl" Oct 30 12:09:58.676872 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:09:58.676803 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-48ksl" podStartSLOduration=1.909911602 podStartE2EDuration="6.67678693s" totalImagesPullingTime="4.766875328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:09:58.675582507 +0000 UTC m=+949.709720900" watchObservedRunningTime="2026-10-30 12:09:58.67678693 +0000 UTC m=+949.710925335" Oct 30 12:10:04.675986 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:04.675955 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-48ksl" Oct 30 12:10:12.223298 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.223262 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-786557dc8d-qg9mf"] Oct 30 12:10:12.250407 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.250366 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-786557dc8d-qg9mf"] Oct 30 12:10:12.250593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.250472 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-786557dc8d-qg9mf" Oct 30 12:10:12.264593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.264559 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 12:10:12.264593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.264593 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 12:10:12.276401 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.276369 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-s8v5b\"" Oct 30 12:10:12.276539 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.276400 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 12:10:12.276539 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.276443 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 12:10:12.276539 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.276484 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 12:10:12.365120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.365065 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-metrics-certs\") pod \"maas-controller-786557dc8d-qg9mf\" (UID: \"65a1f5cc-c0af-49c1-9d9c-bd708ca437b3\") " pod="opendatahub/maas-controller-786557dc8d-qg9mf" Oct 30 12:10:12.365288 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.365145 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbvdb\" (UniqueName: \"kubernetes.io/projected/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-kube-api-access-pbvdb\") pod \"maas-controller-786557dc8d-qg9mf\" (UID: \"65a1f5cc-c0af-49c1-9d9c-bd708ca437b3\") " pod="opendatahub/maas-controller-786557dc8d-qg9mf" Oct 30 12:10:12.365288 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.365220 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-webhook-cert\") pod \"maas-controller-786557dc8d-qg9mf\" (UID: \"65a1f5cc-c0af-49c1-9d9c-bd708ca437b3\") " pod="opendatahub/maas-controller-786557dc8d-qg9mf" Oct 30 12:10:12.466358 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.466310 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbvdb\" (UniqueName: \"kubernetes.io/projected/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-kube-api-access-pbvdb\") pod \"maas-controller-786557dc8d-qg9mf\" (UID: \"65a1f5cc-c0af-49c1-9d9c-bd708ca437b3\") " pod="opendatahub/maas-controller-786557dc8d-qg9mf" Oct 30 12:10:12.466358 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.466365 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-webhook-cert\") pod \"maas-controller-786557dc8d-qg9mf\" (UID: \"65a1f5cc-c0af-49c1-9d9c-bd708ca437b3\") " pod="opendatahub/maas-controller-786557dc8d-qg9mf" Oct 30 12:10:12.466622 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.466427 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-metrics-certs\") pod \"maas-controller-786557dc8d-qg9mf\" (UID: \"65a1f5cc-c0af-49c1-9d9c-bd708ca437b3\") " pod="opendatahub/maas-controller-786557dc8d-qg9mf" Oct 30 12:10:12.468861 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.468834 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-webhook-cert\") pod \"maas-controller-786557dc8d-qg9mf\" (UID: \"65a1f5cc-c0af-49c1-9d9c-bd708ca437b3\") " pod="opendatahub/maas-controller-786557dc8d-qg9mf" Oct 30 12:10:12.468964 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.468875 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-metrics-certs\") pod \"maas-controller-786557dc8d-qg9mf\" (UID: \"65a1f5cc-c0af-49c1-9d9c-bd708ca437b3\") " pod="opendatahub/maas-controller-786557dc8d-qg9mf" Oct 30 12:10:12.489540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.489476 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbvdb\" (UniqueName: \"kubernetes.io/projected/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-kube-api-access-pbvdb\") pod \"maas-controller-786557dc8d-qg9mf\" (UID: \"65a1f5cc-c0af-49c1-9d9c-bd708ca437b3\") " pod="opendatahub/maas-controller-786557dc8d-qg9mf" Oct 30 12:10:12.564765 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.564721 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-786557dc8d-qg9mf" Oct 30 12:10:12.764970 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.764901 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bb8f9449b-nnd2b"] Oct 30 12:10:12.769403 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.769354 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" Oct 30 12:10:12.788307 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.788275 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bb8f9449b-nnd2b"] Oct 30 12:10:12.860140 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.860118 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-786557dc8d-qg9mf"] Oct 30 12:10:12.861534 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:10:12.861502 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a1f5cc_c0af_49c1_9d9c_bd708ca437b3.slice/crio-b64eed2c57e518acc64ff9b4e4986b160a5a052bbe540d2847745c76e32681c2 WatchSource:0}: Error finding container b64eed2c57e518acc64ff9b4e4986b160a5a052bbe540d2847745c76e32681c2: Status 404 returned error can't find the container with id b64eed2c57e518acc64ff9b4e4986b160a5a052bbe540d2847745c76e32681c2 Oct 30 12:10:12.869836 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.869810 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/731fb32a-31aa-4d57-8edd-4f890b9e9288-metrics-certs\") pod \"maas-controller-7bb8f9449b-nnd2b\" (UID: \"731fb32a-31aa-4d57-8edd-4f890b9e9288\") " pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" Oct 30 12:10:12.869938 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.869843 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76965\" (UniqueName: \"kubernetes.io/projected/731fb32a-31aa-4d57-8edd-4f890b9e9288-kube-api-access-76965\") pod \"maas-controller-7bb8f9449b-nnd2b\" (UID: \"731fb32a-31aa-4d57-8edd-4f890b9e9288\") " pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" Oct 30 12:10:12.870041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.870019 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/731fb32a-31aa-4d57-8edd-4f890b9e9288-webhook-cert\") pod \"maas-controller-7bb8f9449b-nnd2b\" (UID: \"731fb32a-31aa-4d57-8edd-4f890b9e9288\") " pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" Oct 30 12:10:12.971209 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.971174 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/731fb32a-31aa-4d57-8edd-4f890b9e9288-webhook-cert\") pod \"maas-controller-7bb8f9449b-nnd2b\" (UID: \"731fb32a-31aa-4d57-8edd-4f890b9e9288\") " pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" Oct 30 12:10:12.971209 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.971214 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/731fb32a-31aa-4d57-8edd-4f890b9e9288-metrics-certs\") pod \"maas-controller-7bb8f9449b-nnd2b\" (UID: \"731fb32a-31aa-4d57-8edd-4f890b9e9288\") " pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" Oct 30 12:10:12.971417 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.971233 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76965\" (UniqueName: \"kubernetes.io/projected/731fb32a-31aa-4d57-8edd-4f890b9e9288-kube-api-access-76965\") pod \"maas-controller-7bb8f9449b-nnd2b\" (UID: \"731fb32a-31aa-4d57-8edd-4f890b9e9288\") " pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" Oct 30 12:10:12.973680 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.973658 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/731fb32a-31aa-4d57-8edd-4f890b9e9288-metrics-certs\") pod \"maas-controller-7bb8f9449b-nnd2b\" (UID: \"731fb32a-31aa-4d57-8edd-4f890b9e9288\") " pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" Oct 30 12:10:12.973762 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.973699 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/731fb32a-31aa-4d57-8edd-4f890b9e9288-webhook-cert\") pod \"maas-controller-7bb8f9449b-nnd2b\" (UID: \"731fb32a-31aa-4d57-8edd-4f890b9e9288\") " pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" Oct 30 12:10:12.994296 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:12.994262 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76965\" (UniqueName: \"kubernetes.io/projected/731fb32a-31aa-4d57-8edd-4f890b9e9288-kube-api-access-76965\") pod \"maas-controller-7bb8f9449b-nnd2b\" (UID: \"731fb32a-31aa-4d57-8edd-4f890b9e9288\") " pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" Oct 30 12:10:13.082903 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:13.082812 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" Oct 30 12:10:13.379860 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:13.379837 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bb8f9449b-nnd2b"] Oct 30 12:10:13.381762 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:10:13.381723 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731fb32a_31aa_4d57_8edd_4f890b9e9288.slice/crio-92a1c83e7c841a9aa78414bc50ba82509f81ca95f281383d478d42d8c1a75760 WatchSource:0}: Error finding container 92a1c83e7c841a9aa78414bc50ba82509f81ca95f281383d478d42d8c1a75760: Status 404 returned error can't find the container with id 92a1c83e7c841a9aa78414bc50ba82509f81ca95f281383d478d42d8c1a75760 Oct 30 12:10:13.704053 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:13.704005 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" event={"ID":"731fb32a-31aa-4d57-8edd-4f890b9e9288","Type":"ContainerStarted","Data":"92a1c83e7c841a9aa78414bc50ba82509f81ca95f281383d478d42d8c1a75760"} Oct 30 12:10:13.705338 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:13.705308 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-786557dc8d-qg9mf" event={"ID":"65a1f5cc-c0af-49c1-9d9c-bd708ca437b3","Type":"ContainerStarted","Data":"b64eed2c57e518acc64ff9b4e4986b160a5a052bbe540d2847745c76e32681c2"} Oct 30 12:10:15.717578 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:15.717545 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-786557dc8d-qg9mf" event={"ID":"65a1f5cc-c0af-49c1-9d9c-bd708ca437b3","Type":"ContainerStarted","Data":"0c11e217155aa02a249864789a749dd26f861896e2786be20f4382da113f254f"} Oct 30 12:10:15.718096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:15.717630 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-786557dc8d-qg9mf" Oct 30 12:10:15.719410 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:15.719385 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" event={"ID":"731fb32a-31aa-4d57-8edd-4f890b9e9288","Type":"ContainerStarted","Data":"022a351b09d8e75d8de1d71134f12cd630c9970870fc1eb5670a59e114c40232"} Oct 30 12:10:15.719527 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:15.719514 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" Oct 30 12:10:15.762393 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:15.762271 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-786557dc8d-qg9mf" podStartSLOduration=1.117072096 podStartE2EDuration="3.762254874s" totalImagesPullingTime="2.645182778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:10:15.761162568 +0000 UTC m=+966.795300964" watchObservedRunningTime="2026-10-30 12:10:15.762254874 +0000 UTC m=+966.796393268" Oct 30 12:10:15.802604 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:15.802550 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" podStartSLOduration=1.670088185 podStartE2EDuration="3.802535235s" totalImagesPullingTime="2.13244705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:10:15.802445853 +0000 UTC m=+966.836584249" watchObservedRunningTime="2026-10-30 12:10:15.802535235 +0000 UTC m=+966.836673630" Oct 30 12:10:17.683641 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.683606 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs"] Oct 30 12:10:17.687174 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.687154 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs" Oct 30 12:10:17.702464 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.702434 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 12:10:17.712902 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.712876 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-mlkzp\"" Oct 30 12:10:17.713037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.712975 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 12:10:17.714976 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.714954 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs"] Oct 30 12:10:17.715175 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.715159 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 12:10:17.813534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.813492 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdg7r\" (UniqueName: \"kubernetes.io/projected/fb5b969c-76f1-47a1-8fc0-00fac17f17d0-kube-api-access-xdg7r\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-j7nfs\" (UID: \"fb5b969c-76f1-47a1-8fc0-00fac17f17d0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs" Oct 30 12:10:17.813709 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.813541 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5b969c-76f1-47a1-8fc0-00fac17f17d0-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-j7nfs\" (UID: \"fb5b969c-76f1-47a1-8fc0-00fac17f17d0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs" Oct 30 12:10:17.813709 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.813662 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/fb5b969c-76f1-47a1-8fc0-00fac17f17d0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-j7nfs\" (UID: \"fb5b969c-76f1-47a1-8fc0-00fac17f17d0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs" Oct 30 12:10:17.813709 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.813694 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb5b969c-76f1-47a1-8fc0-00fac17f17d0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-j7nfs\" (UID: \"fb5b969c-76f1-47a1-8fc0-00fac17f17d0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs" Oct 30 12:10:17.914939 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.914901 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdg7r\" (UniqueName: \"kubernetes.io/projected/fb5b969c-76f1-47a1-8fc0-00fac17f17d0-kube-api-access-xdg7r\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-j7nfs\" (UID: \"fb5b969c-76f1-47a1-8fc0-00fac17f17d0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs" Oct 30 12:10:17.914939 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.914947 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5b969c-76f1-47a1-8fc0-00fac17f17d0-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-j7nfs\" (UID: \"fb5b969c-76f1-47a1-8fc0-00fac17f17d0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs" Oct 30 12:10:17.915203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.915002 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/fb5b969c-76f1-47a1-8fc0-00fac17f17d0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-j7nfs\" (UID: \"fb5b969c-76f1-47a1-8fc0-00fac17f17d0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs" Oct 30 12:10:17.915203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.915028 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb5b969c-76f1-47a1-8fc0-00fac17f17d0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-j7nfs\" (UID: \"fb5b969c-76f1-47a1-8fc0-00fac17f17d0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs" Oct 30 12:10:17.915760 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.915738 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5b969c-76f1-47a1-8fc0-00fac17f17d0-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-j7nfs\" (UID: \"fb5b969c-76f1-47a1-8fc0-00fac17f17d0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs" Oct 30 12:10:17.915890 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.915869 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/fb5b969c-76f1-47a1-8fc0-00fac17f17d0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-j7nfs\" (UID: \"fb5b969c-76f1-47a1-8fc0-00fac17f17d0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs" Oct 30 12:10:17.917535 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.917518 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb5b969c-76f1-47a1-8fc0-00fac17f17d0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-j7nfs\" (UID: \"fb5b969c-76f1-47a1-8fc0-00fac17f17d0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs" Oct 30 12:10:17.937619 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.937553 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdg7r\" (UniqueName: \"kubernetes.io/projected/fb5b969c-76f1-47a1-8fc0-00fac17f17d0-kube-api-access-xdg7r\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-j7nfs\" (UID: \"fb5b969c-76f1-47a1-8fc0-00fac17f17d0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs" Oct 30 12:10:17.999265 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:17.999227 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs" Oct 30 12:10:18.296995 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:18.296968 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs"] Oct 30 12:10:18.297373 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:10:18.297351 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5b969c_76f1_47a1_8fc0_00fac17f17d0.slice/crio-e7f0eb455d74876ee5890144e5d80a0a493cfd0d67b1446a27d09b9f74250293 WatchSource:0}: Error finding container e7f0eb455d74876ee5890144e5d80a0a493cfd0d67b1446a27d09b9f74250293: Status 404 returned error can't find the container with id e7f0eb455d74876ee5890144e5d80a0a493cfd0d67b1446a27d09b9f74250293 Oct 30 12:10:18.734144 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:18.734108 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs" event={"ID":"fb5b969c-76f1-47a1-8fc0-00fac17f17d0","Type":"ContainerStarted","Data":"e7f0eb455d74876ee5890144e5d80a0a493cfd0d67b1446a27d09b9f74250293"} Oct 30 12:10:23.597609 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:10:23.597568 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:10:26.725647 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:26.725613 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-786557dc8d-qg9mf" Oct 30 12:10:26.726159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:26.725689 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" Oct 30 12:10:26.849241 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:26.849208 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-786557dc8d-qg9mf"] Oct 30 12:10:26.849674 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:26.849646 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-786557dc8d-qg9mf" podUID="65a1f5cc-c0af-49c1-9d9c-bd708ca437b3" containerName="manager" containerID="cri-o://0c11e217155aa02a249864789a749dd26f861896e2786be20f4382da113f254f" gracePeriod=30 Oct 30 12:10:27.129626 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:27.129601 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-786557dc8d-qg9mf" Oct 30 12:10:27.300187 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:27.300095 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-webhook-cert\" (UniqueName: \"kubernetes.io/secret/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-webhook-cert\") pod \"65a1f5cc-c0af-49c1-9d9c-bd708ca437b3\" (UID: \"65a1f5cc-c0af-49c1-9d9c-bd708ca437b3\") " Oct 30 12:10:27.300187 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:27.300180 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-kube-api-access-pbvdb\" (UniqueName: \"kubernetes.io/projected/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-kube-api-access-pbvdb\") pod \"65a1f5cc-c0af-49c1-9d9c-bd708ca437b3\" (UID: \"65a1f5cc-c0af-49c1-9d9c-bd708ca437b3\") " Oct 30 12:10:27.300421 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:27.300203 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-metrics-certs\") pod \"65a1f5cc-c0af-49c1-9d9c-bd708ca437b3\" (UID: \"65a1f5cc-c0af-49c1-9d9c-bd708ca437b3\") " Oct 30 12:10:27.302870 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:27.302833 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-webhook-cert" pod "65a1f5cc-c0af-49c1-9d9c-bd708ca437b3" (UID: "65a1f5cc-c0af-49c1-9d9c-bd708ca437b3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:10:27.303063 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:27.303036 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-metrics-certs" pod "65a1f5cc-c0af-49c1-9d9c-bd708ca437b3" (UID: "65a1f5cc-c0af-49c1-9d9c-bd708ca437b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:10:27.303063 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:27.303047 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-kube-api-access-pbvdb" pod "65a1f5cc-c0af-49c1-9d9c-bd708ca437b3" (UID: "65a1f5cc-c0af-49c1-9d9c-bd708ca437b3"). InnerVolumeSpecName "kube-api-access-pbvdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:10:27.401938 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:27.401889 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbvdb\" (UniqueName: \"kubernetes.io/projected/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-kube-api-access-pbvdb\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:10:27.401938 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:27.401940 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:10:27.402347 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:27.401956 4468 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3-webhook-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:10:27.772848 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:27.772584 4468 generic.go:356] "Generic (PLEG): container finished" podID="65a1f5cc-c0af-49c1-9d9c-bd708ca437b3" containerID="0c11e217155aa02a249864789a749dd26f861896e2786be20f4382da113f254f" exitCode=0 Oct 30 12:10:27.772848 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:27.772635 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-786557dc8d-qg9mf" event={"ID":"65a1f5cc-c0af-49c1-9d9c-bd708ca437b3","Type":"ContainerDied","Data":"0c11e217155aa02a249864789a749dd26f861896e2786be20f4382da113f254f"} Oct 30 12:10:27.772848 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:27.772644 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-786557dc8d-qg9mf" Oct 30 12:10:27.772848 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:27.772673 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-786557dc8d-qg9mf" event={"ID":"65a1f5cc-c0af-49c1-9d9c-bd708ca437b3","Type":"ContainerDied","Data":"b64eed2c57e518acc64ff9b4e4986b160a5a052bbe540d2847745c76e32681c2"} Oct 30 12:10:27.772848 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:27.772705 4468 scope.go:122] "RemoveContainer" containerID="0c11e217155aa02a249864789a749dd26f861896e2786be20f4382da113f254f" Oct 30 12:10:27.789188 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:27.789166 4468 scope.go:122] "RemoveContainer" containerID="0c11e217155aa02a249864789a749dd26f861896e2786be20f4382da113f254f" Oct 30 12:10:27.789447 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:10:27.789428 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c11e217155aa02a249864789a749dd26f861896e2786be20f4382da113f254f\": container with ID starting with 0c11e217155aa02a249864789a749dd26f861896e2786be20f4382da113f254f not found: ID does not exist" containerID="0c11e217155aa02a249864789a749dd26f861896e2786be20f4382da113f254f" Oct 30 12:10:27.789535 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:27.789453 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c11e217155aa02a249864789a749dd26f861896e2786be20f4382da113f254f"} err="failed to get container status \"0c11e217155aa02a249864789a749dd26f861896e2786be20f4382da113f254f\": rpc error: code = NotFound desc = could not find container \"0c11e217155aa02a249864789a749dd26f861896e2786be20f4382da113f254f\": container with ID starting with 0c11e217155aa02a249864789a749dd26f861896e2786be20f4382da113f254f not found: ID does not exist" Oct 30 12:10:27.821690 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:27.821645 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-786557dc8d-qg9mf"] Oct 30 12:10:27.825192 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:27.825142 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-786557dc8d-qg9mf"] Oct 30 12:10:29.088032 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:29.087995 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65a1f5cc-c0af-49c1-9d9c-bd708ca437b3" path="/var/lib/kubelet/pods/65a1f5cc-c0af-49c1-9d9c-bd708ca437b3/volumes" Oct 30 12:10:34.805275 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:34.805226 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs" event={"ID":"fb5b969c-76f1-47a1-8fc0-00fac17f17d0","Type":"ContainerStarted","Data":"f844b230a7a6eb4bf801aca26726fe839b0f960d61ad011c6dcb666c3c7dfab2"} Oct 30 12:10:34.805665 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:34.805361 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs" Oct 30 12:10:34.846107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:34.846045 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs" podStartSLOduration=1.7495829779999998 podStartE2EDuration="17.84603265s" totalImagesPullingTime="16.096449672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:10:34.845013815 +0000 UTC m=+985.879152211" watchObservedRunningTime="2026-10-30 12:10:34.84603265 +0000 UTC m=+985.880171079" Oct 30 12:10:45.814737 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:45.814704 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-j7nfs" Oct 30 12:10:50.210835 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.210797 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-868774c7cf-66wv8"] Oct 30 12:10:50.211440 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.211271 4468 cpu_manager.go:414] "containerMap: removing container" podUID="65a1f5cc-c0af-49c1-9d9c-bd708ca437b3" containerName="manager" Oct 30 12:10:50.211440 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.211290 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65a1f5cc-c0af-49c1-9d9c-bd708ca437b3" containerName="manager" Oct 30 12:10:50.211440 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.211376 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="65a1f5cc-c0af-49c1-9d9c-bd708ca437b3" containerName="manager" Oct 30 12:10:50.215795 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.215774 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" Oct 30 12:10:50.235208 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.235182 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-868774c7cf-66wv8"] Oct 30 12:10:50.245123 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.245101 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rtvpm\"" Oct 30 12:10:50.247114 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.247095 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 12:10:50.247376 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.247356 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 12:10:50.280130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.280099 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b7e442-b7e8-4493-9fd7-50810eda3a6b-openshift-service-ca\") pod \"maas-api-868774c7cf-66wv8\" (UID: \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" Oct 30 12:10:50.280248 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.280140 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b7e442-b7e8-4493-9fd7-50810eda3a6b-metrics-certs\") pod \"maas-api-868774c7cf-66wv8\" (UID: \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" Oct 30 12:10:50.280248 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.280160 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhlnv\" (UniqueName: \"kubernetes.io/projected/43b7e442-b7e8-4493-9fd7-50810eda3a6b-kube-api-access-qhlnv\") pod \"maas-api-868774c7cf-66wv8\" (UID: \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" Oct 30 12:10:50.280342 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.280261 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43b7e442-b7e8-4493-9fd7-50810eda3a6b-maas-api-tls\") pod \"maas-api-868774c7cf-66wv8\" (UID: \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" Oct 30 12:10:50.381230 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.381200 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b7e442-b7e8-4493-9fd7-50810eda3a6b-openshift-service-ca\") pod \"maas-api-868774c7cf-66wv8\" (UID: \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" Oct 30 12:10:50.381366 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.381251 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b7e442-b7e8-4493-9fd7-50810eda3a6b-metrics-certs\") pod \"maas-api-868774c7cf-66wv8\" (UID: \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" Oct 30 12:10:50.381366 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.381282 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhlnv\" (UniqueName: \"kubernetes.io/projected/43b7e442-b7e8-4493-9fd7-50810eda3a6b-kube-api-access-qhlnv\") pod \"maas-api-868774c7cf-66wv8\" (UID: \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" Oct 30 12:10:50.381366 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.381332 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43b7e442-b7e8-4493-9fd7-50810eda3a6b-maas-api-tls\") pod \"maas-api-868774c7cf-66wv8\" (UID: \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" Oct 30 12:10:50.381875 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.381853 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b7e442-b7e8-4493-9fd7-50810eda3a6b-openshift-service-ca\") pod \"maas-api-868774c7cf-66wv8\" (UID: \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" Oct 30 12:10:50.383679 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.383658 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43b7e442-b7e8-4493-9fd7-50810eda3a6b-maas-api-tls\") pod \"maas-api-868774c7cf-66wv8\" (UID: \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" Oct 30 12:10:50.383763 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.383718 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b7e442-b7e8-4493-9fd7-50810eda3a6b-metrics-certs\") pod \"maas-api-868774c7cf-66wv8\" (UID: \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" Oct 30 12:10:50.403905 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.403882 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhlnv\" (UniqueName: \"kubernetes.io/projected/43b7e442-b7e8-4493-9fd7-50810eda3a6b-kube-api-access-qhlnv\") pod \"maas-api-868774c7cf-66wv8\" (UID: \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" Oct 30 12:10:50.527459 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.527395 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" Oct 30 12:10:50.824075 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.824042 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-868774c7cf-66wv8"] Oct 30 12:10:50.825097 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:10:50.825058 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b7e442_b7e8_4493_9fd7_50810eda3a6b.slice/crio-25f8a7ffc1e700049bd1d5c11628d906508f136e5033c94009a200c6c112da05 WatchSource:0}: Error finding container 25f8a7ffc1e700049bd1d5c11628d906508f136e5033c94009a200c6c112da05: Status 404 returned error can't find the container with id 25f8a7ffc1e700049bd1d5c11628d906508f136e5033c94009a200c6c112da05 Oct 30 12:10:50.867213 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:50.867183 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" event={"ID":"43b7e442-b7e8-4493-9fd7-50810eda3a6b","Type":"ContainerStarted","Data":"25f8a7ffc1e700049bd1d5c11628d906508f136e5033c94009a200c6c112da05"} Oct 30 12:10:52.877064 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:52.876985 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" event={"ID":"43b7e442-b7e8-4493-9fd7-50810eda3a6b","Type":"ContainerStarted","Data":"b1eb330fc0e62e156a9369a96b51295cb32e1d043f5f62a2b57b5f76097980a1"} Oct 30 12:10:52.877532 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:52.877109 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" Oct 30 12:10:52.917864 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:52.917815 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" podStartSLOduration=1.245372307 podStartE2EDuration="2.91780155s" totalImagesPullingTime="1.672429243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:10:52.917051718 +0000 UTC m=+1003.951190113" watchObservedRunningTime="2026-10-30 12:10:52.91780155 +0000 UTC m=+1003.951939995" Oct 30 12:10:53.596765 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:10:53.596730 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:10:58.885945 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:58.885917 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" Oct 30 12:10:59.701253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:59.701217 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-l8mr9"] Oct 30 12:10:59.708018 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:59.707999 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-l8mr9" Oct 30 12:10:59.721313 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:59.721291 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-l8mr9"] Oct 30 12:10:59.756023 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:59.755997 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjl9x\" (UniqueName: \"kubernetes.io/projected/a31b587f-8706-4c9b-b556-be2372b7ff88-kube-api-access-xjl9x\") pod \"authorino-6f6944bcdd-l8mr9\" (UID: \"a31b587f-8706-4c9b-b556-be2372b7ff88\") " pod="kuadrant-system/authorino-6f6944bcdd-l8mr9" Oct 30 12:10:59.857399 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:59.857361 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjl9x\" (UniqueName: \"kubernetes.io/projected/a31b587f-8706-4c9b-b556-be2372b7ff88-kube-api-access-xjl9x\") pod \"authorino-6f6944bcdd-l8mr9\" (UID: \"a31b587f-8706-4c9b-b556-be2372b7ff88\") " pod="kuadrant-system/authorino-6f6944bcdd-l8mr9" Oct 30 12:10:59.878604 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:59.878577 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-l8mr9"] Oct 30 12:10:59.878768 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:10:59.878749 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xjl9x], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-l8mr9" podUID="a31b587f-8706-4c9b-b556-be2372b7ff88" Oct 30 12:10:59.884634 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:59.884610 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjl9x\" (UniqueName: \"kubernetes.io/projected/a31b587f-8706-4c9b-b556-be2372b7ff88-kube-api-access-xjl9x\") pod \"authorino-6f6944bcdd-l8mr9\" (UID: \"a31b587f-8706-4c9b-b556-be2372b7ff88\") " pod="kuadrant-system/authorino-6f6944bcdd-l8mr9" Oct 30 12:10:59.904521 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:59.904493 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-l8mr9" Oct 30 12:10:59.910425 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:59.910400 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-lmj92"] Oct 30 12:10:59.910834 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:59.910817 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-l8mr9" Oct 30 12:10:59.912235 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:59.912219 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-lmj92" Oct 30 12:10:59.926595 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:59.926577 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 12:10:59.934319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:59.934298 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-lmj92"] Oct 30 12:10:59.958062 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:59.957999 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a31b587f-8706-4c9b-b556-be2372b7ff88-kube-api-access-xjl9x\" (UniqueName: \"kubernetes.io/projected/a31b587f-8706-4c9b-b556-be2372b7ff88-kube-api-access-xjl9x\") pod \"a31b587f-8706-4c9b-b556-be2372b7ff88\" (UID: \"a31b587f-8706-4c9b-b556-be2372b7ff88\") " Oct 30 12:10:59.958180 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:59.958139 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/47c668bd-6892-465f-8c35-a9350e29271d-tls-cert\") pod \"authorino-6889cbd8c4-lmj92\" (UID: \"47c668bd-6892-465f-8c35-a9350e29271d\") " pod="kuadrant-system/authorino-6889cbd8c4-lmj92" Oct 30 12:10:59.958180 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:59.958172 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjhd5\" (UniqueName: \"kubernetes.io/projected/47c668bd-6892-465f-8c35-a9350e29271d-kube-api-access-wjhd5\") pod \"authorino-6889cbd8c4-lmj92\" (UID: \"47c668bd-6892-465f-8c35-a9350e29271d\") " pod="kuadrant-system/authorino-6889cbd8c4-lmj92" Oct 30 12:10:59.960234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:10:59.960208 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31b587f-8706-4c9b-b556-be2372b7ff88-kube-api-access-xjl9x" pod "a31b587f-8706-4c9b-b556-be2372b7ff88" (UID: "a31b587f-8706-4c9b-b556-be2372b7ff88"). InnerVolumeSpecName "kube-api-access-xjl9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:11:00.059010 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.058977 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjhd5\" (UniqueName: \"kubernetes.io/projected/47c668bd-6892-465f-8c35-a9350e29271d-kube-api-access-wjhd5\") pod \"authorino-6889cbd8c4-lmj92\" (UID: \"47c668bd-6892-465f-8c35-a9350e29271d\") " pod="kuadrant-system/authorino-6889cbd8c4-lmj92" Oct 30 12:11:00.059193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.059055 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/47c668bd-6892-465f-8c35-a9350e29271d-tls-cert\") pod \"authorino-6889cbd8c4-lmj92\" (UID: \"47c668bd-6892-465f-8c35-a9350e29271d\") " pod="kuadrant-system/authorino-6889cbd8c4-lmj92" Oct 30 12:11:00.059193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.059105 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjl9x\" (UniqueName: \"kubernetes.io/projected/a31b587f-8706-4c9b-b556-be2372b7ff88-kube-api-access-xjl9x\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:11:00.061420 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.061393 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/47c668bd-6892-465f-8c35-a9350e29271d-tls-cert\") pod \"authorino-6889cbd8c4-lmj92\" (UID: \"47c668bd-6892-465f-8c35-a9350e29271d\") " pod="kuadrant-system/authorino-6889cbd8c4-lmj92" Oct 30 12:11:00.083971 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.083946 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjhd5\" (UniqueName: \"kubernetes.io/projected/47c668bd-6892-465f-8c35-a9350e29271d-kube-api-access-wjhd5\") pod \"authorino-6889cbd8c4-lmj92\" (UID: \"47c668bd-6892-465f-8c35-a9350e29271d\") " pod="kuadrant-system/authorino-6889cbd8c4-lmj92" Oct 30 12:11:00.091441 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.091419 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-lmj92"] Oct 30 12:11:00.091603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.091593 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-lmj92" Oct 30 12:11:00.120576 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.120546 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f96c4569b-7p5fs"] Oct 30 12:11:00.122807 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.122787 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f96c4569b-7p5fs" Oct 30 12:11:00.142603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.142221 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f96c4569b-7p5fs"] Oct 30 12:11:00.159900 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.159831 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e3fdd246-9b07-47a5-8736-08de39cb55cd-tls-cert\") pod \"authorino-5f96c4569b-7p5fs\" (UID: \"e3fdd246-9b07-47a5-8736-08de39cb55cd\") " pod="kuadrant-system/authorino-5f96c4569b-7p5fs" Oct 30 12:11:00.160037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.159963 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns5v9\" (UniqueName: \"kubernetes.io/projected/e3fdd246-9b07-47a5-8736-08de39cb55cd-kube-api-access-ns5v9\") pod \"authorino-5f96c4569b-7p5fs\" (UID: \"e3fdd246-9b07-47a5-8736-08de39cb55cd\") " pod="kuadrant-system/authorino-5f96c4569b-7p5fs" Oct 30 12:11:00.260942 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.260869 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns5v9\" (UniqueName: \"kubernetes.io/projected/e3fdd246-9b07-47a5-8736-08de39cb55cd-kube-api-access-ns5v9\") pod \"authorino-5f96c4569b-7p5fs\" (UID: \"e3fdd246-9b07-47a5-8736-08de39cb55cd\") " pod="kuadrant-system/authorino-5f96c4569b-7p5fs" Oct 30 12:11:00.260942 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.260920 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e3fdd246-9b07-47a5-8736-08de39cb55cd-tls-cert\") pod \"authorino-5f96c4569b-7p5fs\" (UID: \"e3fdd246-9b07-47a5-8736-08de39cb55cd\") " pod="kuadrant-system/authorino-5f96c4569b-7p5fs" Oct 30 12:11:00.263256 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.263239 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e3fdd246-9b07-47a5-8736-08de39cb55cd-tls-cert\") pod \"authorino-5f96c4569b-7p5fs\" (UID: \"e3fdd246-9b07-47a5-8736-08de39cb55cd\") " pod="kuadrant-system/authorino-5f96c4569b-7p5fs" Oct 30 12:11:00.284219 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.284199 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns5v9\" (UniqueName: \"kubernetes.io/projected/e3fdd246-9b07-47a5-8736-08de39cb55cd-kube-api-access-ns5v9\") pod \"authorino-5f96c4569b-7p5fs\" (UID: \"e3fdd246-9b07-47a5-8736-08de39cb55cd\") " pod="kuadrant-system/authorino-5f96c4569b-7p5fs" Oct 30 12:11:00.385733 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.385711 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-lmj92"] Oct 30 12:11:00.387137 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:11:00.387112 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47c668bd_6892_465f_8c35_a9350e29271d.slice/crio-e3b0b685bbcbeeab60b049f62525b2f942e5edde35911847dcc399c29d2ee36f WatchSource:0}: Error finding container e3b0b685bbcbeeab60b049f62525b2f942e5edde35911847dcc399c29d2ee36f: Status 404 returned error can't find the container with id e3b0b685bbcbeeab60b049f62525b2f942e5edde35911847dcc399c29d2ee36f Oct 30 12:11:00.434383 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.434362 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f96c4569b-7p5fs" Oct 30 12:11:00.726535 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.726507 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f96c4569b-7p5fs"] Oct 30 12:11:00.727618 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:11:00.727592 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3fdd246_9b07_47a5_8736_08de39cb55cd.slice/crio-1d07a4664f07b2e061fbe56a5d50714f353d6d42154d92c19632b70a0e5754f4 WatchSource:0}: Error finding container 1d07a4664f07b2e061fbe56a5d50714f353d6d42154d92c19632b70a0e5754f4: Status 404 returned error can't find the container with id 1d07a4664f07b2e061fbe56a5d50714f353d6d42154d92c19632b70a0e5754f4 Oct 30 12:11:00.909965 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.909927 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-lmj92" event={"ID":"47c668bd-6892-465f-8c35-a9350e29271d","Type":"ContainerStarted","Data":"e3b0b685bbcbeeab60b049f62525b2f942e5edde35911847dcc399c29d2ee36f"} Oct 30 12:11:00.911054 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.911036 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-l8mr9" Oct 30 12:11:00.911054 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.911041 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f96c4569b-7p5fs" event={"ID":"e3fdd246-9b07-47a5-8736-08de39cb55cd","Type":"ContainerStarted","Data":"1d07a4664f07b2e061fbe56a5d50714f353d6d42154d92c19632b70a0e5754f4"} Oct 30 12:11:00.984507 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.984426 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-l8mr9"] Oct 30 12:11:00.985980 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:00.985946 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-l8mr9"] Oct 30 12:11:01.087179 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:01.087154 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a31b587f-8706-4c9b-b556-be2372b7ff88" path="/var/lib/kubelet/pods/a31b587f-8706-4c9b-b556-be2372b7ff88/volumes" Oct 30 12:11:01.917409 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:01.917355 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-lmj92" event={"ID":"47c668bd-6892-465f-8c35-a9350e29271d","Type":"ContainerStarted","Data":"71565cab80a667d3c65005318a9ea58b974885408c4ffacf1e6eacac3f78e57e"} Oct 30 12:11:01.917910 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:01.917449 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-lmj92" podUID="47c668bd-6892-465f-8c35-a9350e29271d" containerName="authorino" containerID="cri-o://71565cab80a667d3c65005318a9ea58b974885408c4ffacf1e6eacac3f78e57e" gracePeriod=30 Oct 30 12:11:01.919507 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:01.919476 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f96c4569b-7p5fs" event={"ID":"e3fdd246-9b07-47a5-8736-08de39cb55cd","Type":"ContainerStarted","Data":"9d588cae9c8b7c852d0a89bda0bb41933a9a5775251de624913414b6e199b393"} Oct 30 12:11:01.958013 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:01.957959 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-lmj92" podStartSLOduration=2.3334886089999998 podStartE2EDuration="2.957945902s" totalImagesPullingTime="624.457293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:01.957780565 +0000 UTC m=+1012.991918961" watchObservedRunningTime="2026-10-30 12:11:01.957945902 +0000 UTC m=+1012.992084298" Oct 30 12:11:01.998625 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:01.998556 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f96c4569b-7p5fs" podStartSLOduration=1.571352898 podStartE2EDuration="1.998535927s" totalImagesPullingTime="427.183029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:01.996183282 +0000 UTC m=+1013.030321757" watchObservedRunningTime="2026-10-30 12:11:01.998535927 +0000 UTC m=+1013.032674325" Oct 30 12:11:02.027845 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.027780 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-vbmqz"] Oct 30 12:11:02.028351 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.028322 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-vbmqz" podUID="38e3670c-3e3a-4e25-a591-a5804c478e5b" containerName="authorino" containerID="cri-o://8d1b0d352a3b2cb73f9895c02510f8c208d08ad4310a2f7bb76aba8ff121d51a" gracePeriod=30 Oct 30 12:11:02.189520 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.189489 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-lmj92" Oct 30 12:11:02.280394 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.280344 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-vbmqz" Oct 30 12:11:02.282177 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.282156 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c668bd-6892-465f-8c35-a9350e29271d-kube-api-access-wjhd5\" (UniqueName: \"kubernetes.io/projected/47c668bd-6892-465f-8c35-a9350e29271d-kube-api-access-wjhd5\") pod \"47c668bd-6892-465f-8c35-a9350e29271d\" (UID: \"47c668bd-6892-465f-8c35-a9350e29271d\") " Oct 30 12:11:02.282301 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.282212 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47c668bd-6892-465f-8c35-a9350e29271d-tls-cert\" (UniqueName: \"kubernetes.io/secret/47c668bd-6892-465f-8c35-a9350e29271d-tls-cert\") pod \"47c668bd-6892-465f-8c35-a9350e29271d\" (UID: \"47c668bd-6892-465f-8c35-a9350e29271d\") " Oct 30 12:11:02.284294 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.284268 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c668bd-6892-465f-8c35-a9350e29271d-kube-api-access-wjhd5" pod "47c668bd-6892-465f-8c35-a9350e29271d" (UID: "47c668bd-6892-465f-8c35-a9350e29271d"). InnerVolumeSpecName "kube-api-access-wjhd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:11:02.292476 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.292447 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47c668bd-6892-465f-8c35-a9350e29271d-tls-cert" pod "47c668bd-6892-465f-8c35-a9350e29271d" (UID: "47c668bd-6892-465f-8c35-a9350e29271d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:11:02.383380 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.383341 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e3670c-3e3a-4e25-a591-a5804c478e5b-kube-api-access-4kxxp\" (UniqueName: \"kubernetes.io/projected/38e3670c-3e3a-4e25-a591-a5804c478e5b-kube-api-access-4kxxp\") pod \"38e3670c-3e3a-4e25-a591-a5804c478e5b\" (UID: \"38e3670c-3e3a-4e25-a591-a5804c478e5b\") " Oct 30 12:11:02.383586 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.383546 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjhd5\" (UniqueName: \"kubernetes.io/projected/47c668bd-6892-465f-8c35-a9350e29271d-kube-api-access-wjhd5\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:11:02.383586 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.383567 4468 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/47c668bd-6892-465f-8c35-a9350e29271d-tls-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:11:02.385453 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.385431 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e3670c-3e3a-4e25-a591-a5804c478e5b-kube-api-access-4kxxp" pod "38e3670c-3e3a-4e25-a591-a5804c478e5b" (UID: "38e3670c-3e3a-4e25-a591-a5804c478e5b"). InnerVolumeSpecName "kube-api-access-4kxxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:11:02.483980 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.483906 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kxxp\" (UniqueName: \"kubernetes.io/projected/38e3670c-3e3a-4e25-a591-a5804c478e5b-kube-api-access-4kxxp\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:11:02.925080 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.925043 4468 generic.go:356] "Generic (PLEG): container finished" podID="47c668bd-6892-465f-8c35-a9350e29271d" containerID="71565cab80a667d3c65005318a9ea58b974885408c4ffacf1e6eacac3f78e57e" exitCode=0 Oct 30 12:11:02.925530 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.925110 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-lmj92" Oct 30 12:11:02.925530 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.925123 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-lmj92" event={"ID":"47c668bd-6892-465f-8c35-a9350e29271d","Type":"ContainerDied","Data":"71565cab80a667d3c65005318a9ea58b974885408c4ffacf1e6eacac3f78e57e"} Oct 30 12:11:02.925530 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.925162 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-lmj92" event={"ID":"47c668bd-6892-465f-8c35-a9350e29271d","Type":"ContainerDied","Data":"e3b0b685bbcbeeab60b049f62525b2f942e5edde35911847dcc399c29d2ee36f"} Oct 30 12:11:02.925530 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.925179 4468 scope.go:122] "RemoveContainer" containerID="71565cab80a667d3c65005318a9ea58b974885408c4ffacf1e6eacac3f78e57e" Oct 30 12:11:02.926498 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.926468 4468 generic.go:356] "Generic (PLEG): container finished" podID="38e3670c-3e3a-4e25-a591-a5804c478e5b" containerID="8d1b0d352a3b2cb73f9895c02510f8c208d08ad4310a2f7bb76aba8ff121d51a" exitCode=0 Oct 30 12:11:02.927079 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.927050 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-vbmqz" Oct 30 12:11:02.933410 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.933130 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-vbmqz" event={"ID":"38e3670c-3e3a-4e25-a591-a5804c478e5b","Type":"ContainerDied","Data":"8d1b0d352a3b2cb73f9895c02510f8c208d08ad4310a2f7bb76aba8ff121d51a"} Oct 30 12:11:02.933410 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.933172 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-vbmqz" event={"ID":"38e3670c-3e3a-4e25-a591-a5804c478e5b","Type":"ContainerDied","Data":"b60a5cb970af8cf838c4048aa2f1e46d981d3b02c5fd62ae34a21af46bcc2000"} Oct 30 12:11:02.933615 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.933456 4468 scope.go:122] "RemoveContainer" containerID="71565cab80a667d3c65005318a9ea58b974885408c4ffacf1e6eacac3f78e57e" Oct 30 12:11:02.933729 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:11:02.933710 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71565cab80a667d3c65005318a9ea58b974885408c4ffacf1e6eacac3f78e57e\": container with ID starting with 71565cab80a667d3c65005318a9ea58b974885408c4ffacf1e6eacac3f78e57e not found: ID does not exist" containerID="71565cab80a667d3c65005318a9ea58b974885408c4ffacf1e6eacac3f78e57e" Oct 30 12:11:02.933797 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.933743 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71565cab80a667d3c65005318a9ea58b974885408c4ffacf1e6eacac3f78e57e"} err="failed to get container status \"71565cab80a667d3c65005318a9ea58b974885408c4ffacf1e6eacac3f78e57e\": rpc error: code = NotFound desc = could not find container \"71565cab80a667d3c65005318a9ea58b974885408c4ffacf1e6eacac3f78e57e\": container with ID starting with 71565cab80a667d3c65005318a9ea58b974885408c4ffacf1e6eacac3f78e57e not found: ID does not exist" Oct 30 12:11:02.933797 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.933768 4468 scope.go:122] "RemoveContainer" containerID="8d1b0d352a3b2cb73f9895c02510f8c208d08ad4310a2f7bb76aba8ff121d51a" Oct 30 12:11:02.941424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.941410 4468 scope.go:122] "RemoveContainer" containerID="8d1b0d352a3b2cb73f9895c02510f8c208d08ad4310a2f7bb76aba8ff121d51a" Oct 30 12:11:02.941658 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:11:02.941643 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d1b0d352a3b2cb73f9895c02510f8c208d08ad4310a2f7bb76aba8ff121d51a\": container with ID starting with 8d1b0d352a3b2cb73f9895c02510f8c208d08ad4310a2f7bb76aba8ff121d51a not found: ID does not exist" containerID="8d1b0d352a3b2cb73f9895c02510f8c208d08ad4310a2f7bb76aba8ff121d51a" Oct 30 12:11:02.941736 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.941666 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d1b0d352a3b2cb73f9895c02510f8c208d08ad4310a2f7bb76aba8ff121d51a"} err="failed to get container status \"8d1b0d352a3b2cb73f9895c02510f8c208d08ad4310a2f7bb76aba8ff121d51a\": rpc error: code = NotFound desc = could not find container \"8d1b0d352a3b2cb73f9895c02510f8c208d08ad4310a2f7bb76aba8ff121d51a\": container with ID starting with 8d1b0d352a3b2cb73f9895c02510f8c208d08ad4310a2f7bb76aba8ff121d51a not found: ID does not exist" Oct 30 12:11:02.978494 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.978468 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-lmj92"] Oct 30 12:11:02.990625 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:02.990594 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-lmj92"] Oct 30 12:11:03.022256 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:03.022225 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-vbmqz"] Oct 30 12:11:03.024166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:03.024145 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-vbmqz"] Oct 30 12:11:03.086367 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:03.086338 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e3670c-3e3a-4e25-a591-a5804c478e5b" path="/var/lib/kubelet/pods/38e3670c-3e3a-4e25-a591-a5804c478e5b/volumes" Oct 30 12:11:03.086691 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:03.086679 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47c668bd-6892-465f-8c35-a9350e29271d" path="/var/lib/kubelet/pods/47c668bd-6892-465f-8c35-a9350e29271d/volumes" Oct 30 12:11:09.457580 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:09.457507 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-868774c7cf-66wv8"] Oct 30 12:11:09.458055 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:09.457718 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" podUID="43b7e442-b7e8-4493-9fd7-50810eda3a6b" containerName="maas-api" containerID="cri-o://b1eb330fc0e62e156a9369a96b51295cb32e1d043f5f62a2b57b5f76097980a1" gracePeriod=30 Oct 30 12:11:09.958440 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:09.958406 4468 generic.go:356] "Generic (PLEG): container finished" podID="43b7e442-b7e8-4493-9fd7-50810eda3a6b" containerID="b1eb330fc0e62e156a9369a96b51295cb32e1d043f5f62a2b57b5f76097980a1" exitCode=0 Oct 30 12:11:09.958599 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:09.958454 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" event={"ID":"43b7e442-b7e8-4493-9fd7-50810eda3a6b","Type":"ContainerDied","Data":"b1eb330fc0e62e156a9369a96b51295cb32e1d043f5f62a2b57b5f76097980a1"} Oct 30 12:11:10.191277 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:10.191255 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" Oct 30 12:11:10.248945 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:10.248916 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b7e442-b7e8-4493-9fd7-50810eda3a6b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b7e442-b7e8-4493-9fd7-50810eda3a6b-openshift-service-ca\") pod \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\" (UID: \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\") " Oct 30 12:11:10.249152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:10.248982 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43b7e442-b7e8-4493-9fd7-50810eda3a6b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43b7e442-b7e8-4493-9fd7-50810eda3a6b-maas-api-tls\") pod \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\" (UID: \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\") " Oct 30 12:11:10.249152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:10.249035 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43b7e442-b7e8-4493-9fd7-50810eda3a6b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b7e442-b7e8-4493-9fd7-50810eda3a6b-metrics-certs\") pod \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\" (UID: \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\") " Oct 30 12:11:10.249152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:10.249058 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b7e442-b7e8-4493-9fd7-50810eda3a6b-kube-api-access-qhlnv\" (UniqueName: \"kubernetes.io/projected/43b7e442-b7e8-4493-9fd7-50810eda3a6b-kube-api-access-qhlnv\") pod \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\" (UID: \"43b7e442-b7e8-4493-9fd7-50810eda3a6b\") " Oct 30 12:11:10.249342 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:10.249316 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b7e442-b7e8-4493-9fd7-50810eda3a6b-openshift-service-ca" pod "43b7e442-b7e8-4493-9fd7-50810eda3a6b" (UID: "43b7e442-b7e8-4493-9fd7-50810eda3a6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:11:10.251250 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:10.251225 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b7e442-b7e8-4493-9fd7-50810eda3a6b-kube-api-access-qhlnv" pod "43b7e442-b7e8-4493-9fd7-50810eda3a6b" (UID: "43b7e442-b7e8-4493-9fd7-50810eda3a6b"). InnerVolumeSpecName "kube-api-access-qhlnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:11:10.251250 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:10.251234 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b7e442-b7e8-4493-9fd7-50810eda3a6b-maas-api-tls" pod "43b7e442-b7e8-4493-9fd7-50810eda3a6b" (UID: "43b7e442-b7e8-4493-9fd7-50810eda3a6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:11:10.251390 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:10.251286 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b7e442-b7e8-4493-9fd7-50810eda3a6b-metrics-certs" pod "43b7e442-b7e8-4493-9fd7-50810eda3a6b" (UID: "43b7e442-b7e8-4493-9fd7-50810eda3a6b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:11:10.350377 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:10.350303 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b7e442-b7e8-4493-9fd7-50810eda3a6b-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:11:10.350377 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:10.350334 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhlnv\" (UniqueName: \"kubernetes.io/projected/43b7e442-b7e8-4493-9fd7-50810eda3a6b-kube-api-access-qhlnv\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:11:10.350377 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:10.350348 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b7e442-b7e8-4493-9fd7-50810eda3a6b-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:11:10.350377 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:10.350359 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43b7e442-b7e8-4493-9fd7-50810eda3a6b-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:11:10.963758 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:10.963732 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" Oct 30 12:11:10.963758 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:10.963744 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-868774c7cf-66wv8" event={"ID":"43b7e442-b7e8-4493-9fd7-50810eda3a6b","Type":"ContainerDied","Data":"25f8a7ffc1e700049bd1d5c11628d906508f136e5033c94009a200c6c112da05"} Oct 30 12:11:10.964314 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:10.963797 4468 scope.go:122] "RemoveContainer" containerID="b1eb330fc0e62e156a9369a96b51295cb32e1d043f5f62a2b57b5f76097980a1" Oct 30 12:11:11.013216 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:11.013189 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-868774c7cf-66wv8"] Oct 30 12:11:11.024891 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:11.024866 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-868774c7cf-66wv8"] Oct 30 12:11:11.087157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:11.087131 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b7e442-b7e8-4493-9fd7-50810eda3a6b" path="/var/lib/kubelet/pods/43b7e442-b7e8-4493-9fd7-50810eda3a6b/volumes" Oct 30 12:11:23.597653 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:11:23.597566 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:11:25.132814 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.132782 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t"] Oct 30 12:11:25.133322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.133124 4468 cpu_manager.go:414] "containerMap: removing container" podUID="43b7e442-b7e8-4493-9fd7-50810eda3a6b" containerName="maas-api" Oct 30 12:11:25.133322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.133137 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b7e442-b7e8-4493-9fd7-50810eda3a6b" containerName="maas-api" Oct 30 12:11:25.133322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.133151 4468 cpu_manager.go:414] "containerMap: removing container" podUID="47c668bd-6892-465f-8c35-a9350e29271d" containerName="authorino" Oct 30 12:11:25.133322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.133159 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c668bd-6892-465f-8c35-a9350e29271d" containerName="authorino" Oct 30 12:11:25.133322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.133193 4468 cpu_manager.go:414] "containerMap: removing container" podUID="38e3670c-3e3a-4e25-a591-a5804c478e5b" containerName="authorino" Oct 30 12:11:25.133322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.133199 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e3670c-3e3a-4e25-a591-a5804c478e5b" containerName="authorino" Oct 30 12:11:25.133322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.133261 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b7e442-b7e8-4493-9fd7-50810eda3a6b" containerName="maas-api" Oct 30 12:11:25.133322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.133270 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c668bd-6892-465f-8c35-a9350e29271d" containerName="authorino" Oct 30 12:11:25.133322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.133279 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e3670c-3e3a-4e25-a591-a5804c478e5b" containerName="authorino" Oct 30 12:11:25.137430 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.137414 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.154690 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.154662 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8"] Oct 30 12:11:25.157884 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.157862 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.164766 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.164739 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lkcrv\"" Oct 30 12:11:25.166310 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.166289 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t"] Oct 30 12:11:25.190178 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.190131 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8"] Oct 30 12:11:25.278177 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.278148 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.278177 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.278180 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20bde431-93de-47ab-9cb6-fe90d1a4730a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.278386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.278198 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.278386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.278215 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.278386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.278348 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rclns\" (UniqueName: \"kubernetes.io/projected/20bde431-93de-47ab-9cb6-fe90d1a4730a-kube-api-access-rclns\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.278386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.278383 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.278511 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.278409 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.278544 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.278525 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.278577 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.278544 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.278577 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.278562 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qtwg\" (UniqueName: \"kubernetes.io/projected/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-kube-api-access-9qtwg\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.278639 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.278577 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.278639 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.278592 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.278699 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.278637 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.278731 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.278700 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.278731 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.278725 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.278794 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.278751 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.278794 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.278788 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.279260 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.279222 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.380548 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.380515 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.380548 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.380550 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.380752 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.380583 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.380752 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.380603 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.380752 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.380628 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qtwg\" (UniqueName: \"kubernetes.io/projected/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-kube-api-access-9qtwg\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.380752 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.380653 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.380752 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.380676 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.380752 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.380703 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.380752 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.380730 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.381141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.380756 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.381141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.380794 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.381141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.380828 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.381141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.380852 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.381141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.380902 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.381141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.380929 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20bde431-93de-47ab-9cb6-fe90d1a4730a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.381141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.380954 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.381141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.380979 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.381141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.381018 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rclns\" (UniqueName: \"kubernetes.io/projected/20bde431-93de-47ab-9cb6-fe90d1a4730a-kube-api-access-rclns\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.381141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.381016 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.381141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.381025 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.381141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.381103 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.381141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.381016 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.381794 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.381303 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.381794 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.381330 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.381794 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.381393 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.381794 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.381606 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.381794 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.381768 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20bde431-93de-47ab-9cb6-fe90d1a4730a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.382322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.382292 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.383257 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.383208 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.383344 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.383273 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.383534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.383518 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.383911 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.383895 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.407672 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.407644 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.407802 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.407760 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rclns\" (UniqueName: \"kubernetes.io/projected/20bde431-93de-47ab-9cb6-fe90d1a4730a-kube-api-access-rclns\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.408719 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.408700 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-66j8t\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.413211 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.413181 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qtwg\" (UniqueName: \"kubernetes.io/projected/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-kube-api-access-9qtwg\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rhhh8\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.449904 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.449878 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:25.473548 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.473523 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:25.776619 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.776590 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t"] Oct 30 12:11:25.786628 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:11:25.786600 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20bde431_93de_47ab_9cb6_fe90d1a4730a.slice/crio-fa918b70667379893ef329d4a55256c5ef6b30d0b1d1d3c93763648f414775bd WatchSource:0}: Error finding container fa918b70667379893ef329d4a55256c5ef6b30d0b1d1d3c93763648f414775bd: Status 404 returned error can't find the container with id fa918b70667379893ef329d4a55256c5ef6b30d0b1d1d3c93763648f414775bd Oct 30 12:11:25.816096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:25.816059 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8"] Oct 30 12:11:25.816945 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:11:25.816921 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d42ea5c_ec5b_433f_887a_76d2b8a68f46.slice/crio-5dcd1bcebbc10fad21c94d54c7515781bd7a82c7c0cca3b6f415de6fa25548db WatchSource:0}: Error finding container 5dcd1bcebbc10fad21c94d54c7515781bd7a82c7c0cca3b6f415de6fa25548db: Status 404 returned error can't find the container with id 5dcd1bcebbc10fad21c94d54c7515781bd7a82c7c0cca3b6f415de6fa25548db Oct 30 12:11:26.023354 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:26.023317 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" event={"ID":"20bde431-93de-47ab-9cb6-fe90d1a4730a","Type":"ContainerStarted","Data":"fa918b70667379893ef329d4a55256c5ef6b30d0b1d1d3c93763648f414775bd"} Oct 30 12:11:26.024402 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:26.024377 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" event={"ID":"9d42ea5c-ec5b-433f-887a-76d2b8a68f46","Type":"ContainerStarted","Data":"5dcd1bcebbc10fad21c94d54c7515781bd7a82c7c0cca3b6f415de6fa25548db"} Oct 30 12:11:28.377565 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:28.377537 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:28.377839 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:28.377615 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:28.377839 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:28.377650 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:29.039924 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:29.039892 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" event={"ID":"9d42ea5c-ec5b-433f-887a-76d2b8a68f46","Type":"ContainerStarted","Data":"cf334a47c91ac785d5e84ae7473aae0363d9deae0005b7a90fea4abf2bd946dd"} Oct 30 12:11:29.123525 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:29.123455 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" podStartSLOduration=1.565481454 podStartE2EDuration="4.123432534s" totalImagesPullingTime="2.55795108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:29.094469077 +0000 UTC m=+1040.128607472" watchObservedRunningTime="2026-10-30 12:11:29.123432534 +0000 UTC m=+1040.157570931" Oct 30 12:11:29.473970 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:29.473932 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:29.475415 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:29.475390 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 12:11:29.475476 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:29.475434 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" podUID="9d42ea5c-ec5b-433f-887a-76d2b8a68f46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 12:11:29.960238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:29.960198 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:29.960359 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:29.960288 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:29.960359 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:29.960332 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:30.045757 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:30.045728 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" event={"ID":"20bde431-93de-47ab-9cb6-fe90d1a4730a","Type":"ContainerStarted","Data":"8f1ff5ce81835ec05895069ad6bb73488748542ae8fb7831918bf417428acbdc"} Oct 30 12:11:30.474670 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:30.474629 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 12:11:30.475093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:30.474689 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" podUID="9d42ea5c-ec5b-433f-887a-76d2b8a68f46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 12:11:31.102047 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:31.101995 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" podStartSLOduration=1.931018505 podStartE2EDuration="6.101983658s" totalImagesPullingTime="4.170965153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:31.101590452 +0000 UTC m=+1042.135728847" watchObservedRunningTime="2026-10-30 12:11:31.101983658 +0000 UTC m=+1042.136122054" Oct 30 12:11:31.453116 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:31.452229 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:31.454688 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:31.454647 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-66j8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 12:11:31.454802 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:31.454721 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" podUID="20bde431-93de-47ab-9cb6-fe90d1a4730a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 12:11:31.474135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:31.474108 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 12:11:31.474216 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:31.474153 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" podUID="9d42ea5c-ec5b-433f-887a-76d2b8a68f46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 12:11:32.450492 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:32.450455 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-66j8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 12:11:32.451091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:32.451044 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" podUID="20bde431-93de-47ab-9cb6-fe90d1a4730a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 12:11:32.474429 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:32.474403 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 12:11:32.474625 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:32.474605 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" podUID="9d42ea5c-ec5b-433f-887a-76d2b8a68f46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 12:11:33.450569 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:33.450525 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-66j8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 12:11:33.451232 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:33.451198 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" podUID="20bde431-93de-47ab-9cb6-fe90d1a4730a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 12:11:33.474307 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:33.474271 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 12:11:33.474461 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:33.474315 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" podUID="9d42ea5c-ec5b-433f-887a-76d2b8a68f46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 12:11:34.450719 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:34.450681 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-66j8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 12:11:34.451155 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:34.450753 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" podUID="20bde431-93de-47ab-9cb6-fe90d1a4730a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 12:11:34.479839 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:34.479808 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:34.480319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:34.480290 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:34.481147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:34.481125 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:11:35.450540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:35.450505 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-66j8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 12:11:35.450719 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:35.450560 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" podUID="20bde431-93de-47ab-9cb6-fe90d1a4730a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 12:11:36.454082 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:36.454038 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:36.454570 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:36.454549 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:36.455238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:36.455218 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:11:37.260309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:37.260272 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-46f4f"] Oct 30 12:11:37.263819 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:37.263797 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-46f4f" Oct 30 12:11:37.279600 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:37.279576 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 12:11:37.291172 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:37.291150 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-dx7pr\"" Oct 30 12:11:37.291288 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:37.291260 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 12:11:37.292672 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:37.292650 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-46f4f"] Oct 30 12:11:37.303666 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:37.303643 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71215687-0701-4e7b-b95d-c405f836db83-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-46f4f\" (UID: \"71215687-0701-4e7b-b95d-c405f836db83\") " pod="opendatahub/ai-gateway-controller-7d85675f98-46f4f" Oct 30 12:11:37.303796 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:37.303673 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrc42\" (UniqueName: \"kubernetes.io/projected/71215687-0701-4e7b-b95d-c405f836db83-kube-api-access-rrc42\") pod \"ai-gateway-controller-7d85675f98-46f4f\" (UID: \"71215687-0701-4e7b-b95d-c405f836db83\") " pod="opendatahub/ai-gateway-controller-7d85675f98-46f4f" Oct 30 12:11:37.404227 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:37.404187 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71215687-0701-4e7b-b95d-c405f836db83-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-46f4f\" (UID: \"71215687-0701-4e7b-b95d-c405f836db83\") " pod="opendatahub/ai-gateway-controller-7d85675f98-46f4f" Oct 30 12:11:37.404227 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:37.404229 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrc42\" (UniqueName: \"kubernetes.io/projected/71215687-0701-4e7b-b95d-c405f836db83-kube-api-access-rrc42\") pod \"ai-gateway-controller-7d85675f98-46f4f\" (UID: \"71215687-0701-4e7b-b95d-c405f836db83\") " pod="opendatahub/ai-gateway-controller-7d85675f98-46f4f" Oct 30 12:11:37.406655 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:37.406635 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71215687-0701-4e7b-b95d-c405f836db83-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-46f4f\" (UID: \"71215687-0701-4e7b-b95d-c405f836db83\") " pod="opendatahub/ai-gateway-controller-7d85675f98-46f4f" Oct 30 12:11:37.427096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:37.427063 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrc42\" (UniqueName: \"kubernetes.io/projected/71215687-0701-4e7b-b95d-c405f836db83-kube-api-access-rrc42\") pod \"ai-gateway-controller-7d85675f98-46f4f\" (UID: \"71215687-0701-4e7b-b95d-c405f836db83\") " pod="opendatahub/ai-gateway-controller-7d85675f98-46f4f" Oct 30 12:11:37.575917 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:37.575838 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-46f4f" Oct 30 12:11:37.870821 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:37.870791 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-46f4f"] Oct 30 12:11:37.872147 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:11:37.872121 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71215687_0701_4e7b_b95d_c405f836db83.slice/crio-46cea526bdc7106c706608d5680c5934f402daa7e513ad3330a652de400417ee WatchSource:0}: Error finding container 46cea526bdc7106c706608d5680c5934f402daa7e513ad3330a652de400417ee: Status 404 returned error can't find the container with id 46cea526bdc7106c706608d5680c5934f402daa7e513ad3330a652de400417ee Oct 30 12:11:38.086434 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:38.086395 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-46f4f" event={"ID":"71215687-0701-4e7b-b95d-c405f836db83","Type":"ContainerStarted","Data":"46cea526bdc7106c706608d5680c5934f402daa7e513ad3330a652de400417ee"} Oct 30 12:11:40.095483 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.095449 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-46f4f" event={"ID":"71215687-0701-4e7b-b95d-c405f836db83","Type":"ContainerStarted","Data":"9776260696fbea56b975c9482333a69947a388dd9cf6d3409362cf9f75b7fc83"} Oct 30 12:11:40.095967 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.095581 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-46f4f" Oct 30 12:11:40.135332 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.135286 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-46f4f" podStartSLOduration=1.65878597 podStartE2EDuration="3.135273667s" totalImagesPullingTime="1.476487697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:40.133905548 +0000 UTC m=+1051.168043943" watchObservedRunningTime="2026-10-30 12:11:40.135273667 +0000 UTC m=+1051.169412060" Oct 30 12:11:40.206687 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.206647 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks"] Oct 30 12:11:40.210114 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.210097 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.227485 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.227456 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.227485 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.227484 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.227670 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.227505 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278f2976-5db1-4683-a372-01d103a69b92-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.227670 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.227521 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8hvc\" (UniqueName: \"kubernetes.io/projected/278f2976-5db1-4683-a372-01d103a69b92-kube-api-access-x8hvc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.227670 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.227659 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278f2976-5db1-4683-a372-01d103a69b92-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.227833 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.227696 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.227833 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.227719 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278f2976-5db1-4683-a372-01d103a69b92-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.227833 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.227804 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.228202 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.227860 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.234656 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.234629 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks"] Oct 30 12:11:40.328411 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.328376 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278f2976-5db1-4683-a372-01d103a69b92-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.328411 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.328420 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.328664 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.328449 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.328664 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.328500 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.328664 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.328520 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.328664 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.328535 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278f2976-5db1-4683-a372-01d103a69b92-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.328664 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.328561 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8hvc\" (UniqueName: \"kubernetes.io/projected/278f2976-5db1-4683-a372-01d103a69b92-kube-api-access-x8hvc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.328664 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.328604 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278f2976-5db1-4683-a372-01d103a69b92-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.328664 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.328645 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.329035 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.328945 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.329127 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.329094 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.329127 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.329100 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.329244 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.329203 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.329320 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.329297 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278f2976-5db1-4683-a372-01d103a69b92-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.330846 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.330822 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.331113 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.331089 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278f2976-5db1-4683-a372-01d103a69b92-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.350985 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.350939 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278f2976-5db1-4683-a372-01d103a69b92-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.350985 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.350972 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8hvc\" (UniqueName: \"kubernetes.io/projected/278f2976-5db1-4683-a372-01d103a69b92-kube-api-access-x8hvc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-sbxks\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.522920 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.522889 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:40.833567 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.833542 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks"] Oct 30 12:11:40.834673 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:11:40.834641 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278f2976_5db1_4683_a372_01d103a69b92.slice/crio-26ae04cb39013a617543828b15775094e09e1bf5287a50b3d83fec6259854997 WatchSource:0}: Error finding container 26ae04cb39013a617543828b15775094e09e1bf5287a50b3d83fec6259854997: Status 404 returned error can't find the container with id 26ae04cb39013a617543828b15775094e09e1bf5287a50b3d83fec6259854997 Oct 30 12:11:40.837171 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.837001 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:40.837171 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.837131 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:40.837171 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:40.837172 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:41.101176 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.101090 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" event={"ID":"278f2976-5db1-4683-a372-01d103a69b92","Type":"ContainerStarted","Data":"4828811eda867e3467712d277d943c05bbbd6827ad0f4fc1c82c7b0aca1ea52f"} Oct 30 12:11:41.101176 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.101146 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" event={"ID":"278f2976-5db1-4683-a372-01d103a69b92","Type":"ContainerStarted","Data":"26ae04cb39013a617543828b15775094e09e1bf5287a50b3d83fec6259854997"} Oct 30 12:11:41.145045 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.144992 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" podStartSLOduration=1.144978648 podStartE2EDuration="1.144978648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:41.144821878 +0000 UTC m=+1052.178960284" watchObservedRunningTime="2026-10-30 12:11:41.144978648 +0000 UTC m=+1052.179117043" Oct 30 12:11:41.529256 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.529214 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:41.530545 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.530513 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-sbxks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 12:11:41.530673 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.530575 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" podUID="278f2976-5db1-4683-a372-01d103a69b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 12:11:41.576628 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.576597 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-ftmv4"] Oct 30 12:11:41.580031 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.580011 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-ftmv4" Oct 30 12:11:41.598417 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.598398 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 12:11:41.598707 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.598684 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kv4vr\"" Oct 30 12:11:41.599759 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.599741 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-ftmv4"] Oct 30 12:11:41.634502 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.634471 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-94zmh"] Oct 30 12:11:41.637952 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.637928 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-94zmh" Oct 30 12:11:41.639686 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.639666 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/910664f0-5127-40e1-b2c8-493f050457cc-plugins-config-volume\") pod \"payload-processing-5f78687d5b-ftmv4\" (UID: \"910664f0-5127-40e1-b2c8-493f050457cc\") " pod="openshift-ingress/payload-processing-5f78687d5b-ftmv4" Oct 30 12:11:41.639778 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.639707 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsq8k\" (UniqueName: \"kubernetes.io/projected/910664f0-5127-40e1-b2c8-493f050457cc-kube-api-access-qsq8k\") pod \"payload-processing-5f78687d5b-ftmv4\" (UID: \"910664f0-5127-40e1-b2c8-493f050457cc\") " pod="openshift-ingress/payload-processing-5f78687d5b-ftmv4" Oct 30 12:11:41.656467 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.656445 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-94zmh"] Oct 30 12:11:41.741086 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.740496 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d28f84-07c8-4c39-830b-608b124282f5-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-94zmh\" (UID: \"a1d28f84-07c8-4c39-830b-608b124282f5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-94zmh" Oct 30 12:11:41.741086 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.740558 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm6l7\" (UniqueName: \"kubernetes.io/projected/a1d28f84-07c8-4c39-830b-608b124282f5-kube-api-access-cm6l7\") pod \"payload-pre-processing-7b98bb678c-94zmh\" (UID: \"a1d28f84-07c8-4c39-830b-608b124282f5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-94zmh" Oct 30 12:11:41.741086 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.740661 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/910664f0-5127-40e1-b2c8-493f050457cc-plugins-config-volume\") pod \"payload-processing-5f78687d5b-ftmv4\" (UID: \"910664f0-5127-40e1-b2c8-493f050457cc\") " pod="openshift-ingress/payload-processing-5f78687d5b-ftmv4" Oct 30 12:11:41.741086 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.740727 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsq8k\" (UniqueName: \"kubernetes.io/projected/910664f0-5127-40e1-b2c8-493f050457cc-kube-api-access-qsq8k\") pod \"payload-processing-5f78687d5b-ftmv4\" (UID: \"910664f0-5127-40e1-b2c8-493f050457cc\") " pod="openshift-ingress/payload-processing-5f78687d5b-ftmv4" Oct 30 12:11:41.743088 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.741852 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/910664f0-5127-40e1-b2c8-493f050457cc-plugins-config-volume\") pod \"payload-processing-5f78687d5b-ftmv4\" (UID: \"910664f0-5127-40e1-b2c8-493f050457cc\") " pod="openshift-ingress/payload-processing-5f78687d5b-ftmv4" Oct 30 12:11:41.763610 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.763580 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsq8k\" (UniqueName: \"kubernetes.io/projected/910664f0-5127-40e1-b2c8-493f050457cc-kube-api-access-qsq8k\") pod \"payload-processing-5f78687d5b-ftmv4\" (UID: \"910664f0-5127-40e1-b2c8-493f050457cc\") " pod="openshift-ingress/payload-processing-5f78687d5b-ftmv4" Oct 30 12:11:41.842021 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.841918 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d28f84-07c8-4c39-830b-608b124282f5-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-94zmh\" (UID: \"a1d28f84-07c8-4c39-830b-608b124282f5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-94zmh" Oct 30 12:11:41.842188 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.842027 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm6l7\" (UniqueName: \"kubernetes.io/projected/a1d28f84-07c8-4c39-830b-608b124282f5-kube-api-access-cm6l7\") pod \"payload-pre-processing-7b98bb678c-94zmh\" (UID: \"a1d28f84-07c8-4c39-830b-608b124282f5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-94zmh" Oct 30 12:11:41.842773 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.842748 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d28f84-07c8-4c39-830b-608b124282f5-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-94zmh\" (UID: \"a1d28f84-07c8-4c39-830b-608b124282f5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-94zmh" Oct 30 12:11:41.869094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.866180 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm6l7\" (UniqueName: \"kubernetes.io/projected/a1d28f84-07c8-4c39-830b-608b124282f5-kube-api-access-cm6l7\") pod \"payload-pre-processing-7b98bb678c-94zmh\" (UID: \"a1d28f84-07c8-4c39-830b-608b124282f5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-94zmh" Oct 30 12:11:41.895319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.895213 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-ftmv4" Oct 30 12:11:41.958115 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:41.955219 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-94zmh" Oct 30 12:11:42.280519 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:42.280497 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-ftmv4"] Oct 30 12:11:42.282419 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:11:42.282393 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod910664f0_5127_40e1_b2c8_493f050457cc.slice/crio-cd5de29b074c90b9ef2ffd0b76eddf200a4e43c1602f19e6d4e85fb2be6c1bf5 WatchSource:0}: Error finding container cd5de29b074c90b9ef2ffd0b76eddf200a4e43c1602f19e6d4e85fb2be6c1bf5: Status 404 returned error can't find the container with id cd5de29b074c90b9ef2ffd0b76eddf200a4e43c1602f19e6d4e85fb2be6c1bf5 Oct 30 12:11:42.292808 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:42.292784 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-94zmh"] Oct 30 12:11:42.293415 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:11:42.293394 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d28f84_07c8_4c39_830b_608b124282f5.slice/crio-462a41843ab52dd4e3419ebba9377b5eba2222cea2a9d3adb300dd8ea01bb686 WatchSource:0}: Error finding container 462a41843ab52dd4e3419ebba9377b5eba2222cea2a9d3adb300dd8ea01bb686: Status 404 returned error can't find the container with id 462a41843ab52dd4e3419ebba9377b5eba2222cea2a9d3adb300dd8ea01bb686 Oct 30 12:11:42.528223 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:42.528182 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-sbxks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 12:11:42.528388 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:42.528248 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" podUID="278f2976-5db1-4683-a372-01d103a69b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 12:11:43.143933 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:43.143893 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-94zmh" event={"ID":"a1d28f84-07c8-4c39-830b-608b124282f5","Type":"ContainerStarted","Data":"462a41843ab52dd4e3419ebba9377b5eba2222cea2a9d3adb300dd8ea01bb686"} Oct 30 12:11:43.147799 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:43.147748 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-ftmv4" event={"ID":"910664f0-5127-40e1-b2c8-493f050457cc","Type":"ContainerStarted","Data":"cd5de29b074c90b9ef2ffd0b76eddf200a4e43c1602f19e6d4e85fb2be6c1bf5"} Oct 30 12:11:43.528093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:43.527303 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-sbxks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 12:11:43.528093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:43.527363 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" podUID="278f2976-5db1-4683-a372-01d103a69b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 12:11:44.526963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:44.526488 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-sbxks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 12:11:44.526963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:44.526550 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" podUID="278f2976-5db1-4683-a372-01d103a69b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 12:11:45.532253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:45.531476 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-sbxks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 12:11:45.532253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:45.531532 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" podUID="278f2976-5db1-4683-a372-01d103a69b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 12:11:46.165126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:46.165089 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-94zmh" event={"ID":"a1d28f84-07c8-4c39-830b-608b124282f5","Type":"ContainerStarted","Data":"b5bb35da99a5ee1efb94a7cbff074acb44a92f953a73fc8ae8d4f09cb434a928"} Oct 30 12:11:46.165301 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:46.165154 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-94zmh" Oct 30 12:11:46.166545 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:46.166519 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-ftmv4" event={"ID":"910664f0-5127-40e1-b2c8-493f050457cc","Type":"ContainerStarted","Data":"db6049df5c145941ca18cfbf2a7047ced6dc196767262d7f79a4429fee440b82"} Oct 30 12:11:46.166656 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:46.166637 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-ftmv4" Oct 30 12:11:46.240599 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:46.240544 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-94zmh" podStartSLOduration=2.19908187 podStartE2EDuration="5.240530782s" totalImagesPullingTime="3.041448912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:46.239208886 +0000 UTC m=+1057.273347284" watchObservedRunningTime="2026-10-30 12:11:46.240530782 +0000 UTC m=+1057.274669177" Oct 30 12:11:46.298457 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:46.298409 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-ftmv4" podStartSLOduration=2.089468768 podStartE2EDuration="5.298392608s" totalImagesPullingTime="3.20892384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:46.297850063 +0000 UTC m=+1057.331988459" watchObservedRunningTime="2026-10-30 12:11:46.298392608 +0000 UTC m=+1057.332531004" Oct 30 12:11:46.525976 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:46.525476 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-sbxks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 12:11:46.525976 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:46.525542 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" podUID="278f2976-5db1-4683-a372-01d103a69b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 12:11:47.527292 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:47.527263 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:47.527700 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:47.527530 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:47.528240 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:47.528224 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:11:51.103302 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:51.103273 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-46f4f" Oct 30 12:11:51.397360 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:51.397285 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6858bfbcf8-h9qws"] Oct 30 12:11:51.400923 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:51.400903 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6858bfbcf8-h9qws" Oct 30 12:11:51.419521 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:51.419500 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6858bfbcf8-h9qws"] Oct 30 12:11:51.436341 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:51.436316 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc2e5679-e297-4aae-8676-b55f902809b1-metrics-certs\") pod \"ai-gateway-controller-6858bfbcf8-h9qws\" (UID: \"bc2e5679-e297-4aae-8676-b55f902809b1\") " pod="opendatahub/ai-gateway-controller-6858bfbcf8-h9qws" Oct 30 12:11:51.436447 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:51.436373 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r8lt\" (UniqueName: \"kubernetes.io/projected/bc2e5679-e297-4aae-8676-b55f902809b1-kube-api-access-5r8lt\") pod \"ai-gateway-controller-6858bfbcf8-h9qws\" (UID: \"bc2e5679-e297-4aae-8676-b55f902809b1\") " pod="opendatahub/ai-gateway-controller-6858bfbcf8-h9qws" Oct 30 12:11:51.537750 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:51.537720 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r8lt\" (UniqueName: \"kubernetes.io/projected/bc2e5679-e297-4aae-8676-b55f902809b1-kube-api-access-5r8lt\") pod \"ai-gateway-controller-6858bfbcf8-h9qws\" (UID: \"bc2e5679-e297-4aae-8676-b55f902809b1\") " pod="opendatahub/ai-gateway-controller-6858bfbcf8-h9qws" Oct 30 12:11:51.537916 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:51.537812 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc2e5679-e297-4aae-8676-b55f902809b1-metrics-certs\") pod \"ai-gateway-controller-6858bfbcf8-h9qws\" (UID: \"bc2e5679-e297-4aae-8676-b55f902809b1\") " pod="opendatahub/ai-gateway-controller-6858bfbcf8-h9qws" Oct 30 12:11:51.540169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:51.540143 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc2e5679-e297-4aae-8676-b55f902809b1-metrics-certs\") pod \"ai-gateway-controller-6858bfbcf8-h9qws\" (UID: \"bc2e5679-e297-4aae-8676-b55f902809b1\") " pod="opendatahub/ai-gateway-controller-6858bfbcf8-h9qws" Oct 30 12:11:51.560334 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:51.560309 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r8lt\" (UniqueName: \"kubernetes.io/projected/bc2e5679-e297-4aae-8676-b55f902809b1-kube-api-access-5r8lt\") pod \"ai-gateway-controller-6858bfbcf8-h9qws\" (UID: \"bc2e5679-e297-4aae-8676-b55f902809b1\") " pod="opendatahub/ai-gateway-controller-6858bfbcf8-h9qws" Oct 30 12:11:51.712643 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:51.712613 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6858bfbcf8-h9qws" Oct 30 12:11:52.011094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:52.011052 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6858bfbcf8-h9qws"] Oct 30 12:11:52.012451 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:11:52.012426 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2e5679_e297_4aae_8676_b55f902809b1.slice/crio-7cac47f0201ceb2139fcfbe91338a2a54771f6db13b3285c953951ef430e1799 WatchSource:0}: Error finding container 7cac47f0201ceb2139fcfbe91338a2a54771f6db13b3285c953951ef430e1799: Status 404 returned error can't find the container with id 7cac47f0201ceb2139fcfbe91338a2a54771f6db13b3285c953951ef430e1799 Oct 30 12:11:52.195641 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:52.195608 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6858bfbcf8-h9qws" event={"ID":"bc2e5679-e297-4aae-8676-b55f902809b1","Type":"ContainerStarted","Data":"7cac47f0201ceb2139fcfbe91338a2a54771f6db13b3285c953951ef430e1799"} Oct 30 12:11:53.201429 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:53.201390 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6858bfbcf8-h9qws" event={"ID":"bc2e5679-e297-4aae-8676-b55f902809b1","Type":"ContainerStarted","Data":"ae1ee11cd47629c241b813c17ea6399f01330c32873f1ae94bc674e934ee926c"} Oct 30 12:11:53.201905 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:53.201495 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6858bfbcf8-h9qws" Oct 30 12:11:53.241703 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:53.241651 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6858bfbcf8-h9qws" podStartSLOduration=1.898590393 podStartE2EDuration="2.241637421s" totalImagesPullingTime="343.047028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:53.240044509 +0000 UTC m=+1064.274182906" watchObservedRunningTime="2026-10-30 12:11:53.241637421 +0000 UTC m=+1064.275775816" Oct 30 12:11:53.597205 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:11:53.597132 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:11:55.238851 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.238812 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7"] Oct 30 12:11:55.242464 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.242448 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.268444 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.268411 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/634f6508-78a2-46d9-b94d-3d9a4b184441-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.268630 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.268461 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.268630 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.268492 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.268769 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.268694 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.268769 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.268733 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.268884 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.268820 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.268884 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.268851 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.268884 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.268878 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.269046 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.268898 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk9k8\" (UniqueName: \"kubernetes.io/projected/634f6508-78a2-46d9-b94d-3d9a4b184441-kube-api-access-tk9k8\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.272995 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.272961 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7"] Oct 30 12:11:55.369547 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.369507 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.369547 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.369549 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.369798 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.369610 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.369798 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.369641 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.369798 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.369661 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.369798 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.369680 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9k8\" (UniqueName: \"kubernetes.io/projected/634f6508-78a2-46d9-b94d-3d9a4b184441-kube-api-access-tk9k8\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.369798 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.369712 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/634f6508-78a2-46d9-b94d-3d9a4b184441-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.369798 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.369739 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.369798 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.369769 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.370221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.370189 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.370221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.370209 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.370399 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.370339 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.370472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.370399 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.370677 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.370653 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/634f6508-78a2-46d9-b94d-3d9a4b184441-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.372023 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.372005 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.372258 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.372239 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.392310 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.392280 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.392429 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.392392 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk9k8\" (UniqueName: \"kubernetes.io/projected/634f6508-78a2-46d9-b94d-3d9a4b184441-kube-api-access-tk9k8\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kp7x7\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.555590 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.555493 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:55.868333 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.868311 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7"] Oct 30 12:11:55.869562 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:11:55.869536 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634f6508_78a2_46d9_b94d_3d9a4b184441.slice/crio-f03049407cde8220cdc70cfa7b1d61fb9ea9ac1e3d9cf314e753f5d99138837d WatchSource:0}: Error finding container f03049407cde8220cdc70cfa7b1d61fb9ea9ac1e3d9cf314e753f5d99138837d: Status 404 returned error can't find the container with id f03049407cde8220cdc70cfa7b1d61fb9ea9ac1e3d9cf314e753f5d99138837d Oct 30 12:11:55.871670 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.871640 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:55.871742 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.871702 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:55.871742 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:55.871733 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:11:56.216334 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:56.216298 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" event={"ID":"634f6508-78a2-46d9-b94d-3d9a4b184441","Type":"ContainerStarted","Data":"4bde4b1503dc339fd13fe30cd398b428c8116514c8165b7b1297cc17fb1fbeb7"} Oct 30 12:11:56.216810 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:56.216361 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" event={"ID":"634f6508-78a2-46d9-b94d-3d9a4b184441","Type":"ContainerStarted","Data":"f03049407cde8220cdc70cfa7b1d61fb9ea9ac1e3d9cf314e753f5d99138837d"} Oct 30 12:11:56.279057 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:56.277778 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" podStartSLOduration=1.277759185 podStartE2EDuration="1.277759185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:56.277454922 +0000 UTC m=+1067.311593319" watchObservedRunningTime="2026-10-30 12:11:56.277759185 +0000 UTC m=+1067.311897582" Oct 30 12:11:56.556669 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:56.556580 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:11:56.558250 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:56.558223 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 12:11:56.558394 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:56.558279 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" podUID="634f6508-78a2-46d9-b94d-3d9a4b184441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 12:11:57.173590 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:57.173557 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-94zmh" Oct 30 12:11:57.173762 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:57.173683 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-ftmv4" Oct 30 12:11:57.555950 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:57.555871 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 12:11:57.555950 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:57.555926 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" podUID="634f6508-78a2-46d9-b94d-3d9a4b184441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 12:11:58.556861 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:58.556823 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 12:11:58.557303 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:58.556899 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" podUID="634f6508-78a2-46d9-b94d-3d9a4b184441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 12:11:59.555939 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:59.555895 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 12:11:59.556174 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:11:59.555956 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" podUID="634f6508-78a2-46d9-b94d-3d9a4b184441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 12:12:00.555860 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:00.555825 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 12:12:00.556308 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:00.555881 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" podUID="634f6508-78a2-46d9-b94d-3d9a4b184441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 12:12:01.556897 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:01.556853 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 12:12:01.557317 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:01.556916 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" podUID="634f6508-78a2-46d9-b94d-3d9a4b184441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 12:12:02.560439 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:02.560402 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:12:02.560806 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:02.560693 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:12:02.561319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:02.561302 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:12:04.207169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:04.207138 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6858bfbcf8-h9qws" Oct 30 12:12:04.277264 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:04.277233 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-46f4f"] Oct 30 12:12:04.277464 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:04.277441 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-46f4f" podUID="71215687-0701-4e7b-b95d-c405f836db83" containerName="manager" containerID="cri-o://9776260696fbea56b975c9482333a69947a388dd9cf6d3409362cf9f75b7fc83" gracePeriod=30 Oct 30 12:12:04.536167 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:04.536138 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-46f4f" Oct 30 12:12:04.678814 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:04.678775 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71215687-0701-4e7b-b95d-c405f836db83-kube-api-access-rrc42\" (UniqueName: \"kubernetes.io/projected/71215687-0701-4e7b-b95d-c405f836db83-kube-api-access-rrc42\") pod \"71215687-0701-4e7b-b95d-c405f836db83\" (UID: \"71215687-0701-4e7b-b95d-c405f836db83\") " Oct 30 12:12:04.678988 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:04.678906 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71215687-0701-4e7b-b95d-c405f836db83-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71215687-0701-4e7b-b95d-c405f836db83-metrics-certs\") pod \"71215687-0701-4e7b-b95d-c405f836db83\" (UID: \"71215687-0701-4e7b-b95d-c405f836db83\") " Oct 30 12:12:04.680815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:04.680784 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71215687-0701-4e7b-b95d-c405f836db83-kube-api-access-rrc42" pod "71215687-0701-4e7b-b95d-c405f836db83" (UID: "71215687-0701-4e7b-b95d-c405f836db83"). InnerVolumeSpecName "kube-api-access-rrc42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:12:04.680979 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:04.680962 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71215687-0701-4e7b-b95d-c405f836db83-metrics-certs" pod "71215687-0701-4e7b-b95d-c405f836db83" (UID: "71215687-0701-4e7b-b95d-c405f836db83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:12:04.780390 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:04.780297 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71215687-0701-4e7b-b95d-c405f836db83-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:12:04.780390 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:04.780335 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrc42\" (UniqueName: \"kubernetes.io/projected/71215687-0701-4e7b-b95d-c405f836db83-kube-api-access-rrc42\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:12:05.275831 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:05.270392 4468 generic.go:356] "Generic (PLEG): container finished" podID="71215687-0701-4e7b-b95d-c405f836db83" containerID="9776260696fbea56b975c9482333a69947a388dd9cf6d3409362cf9f75b7fc83" exitCode=0 Oct 30 12:12:05.275831 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:05.270528 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-46f4f" event={"ID":"71215687-0701-4e7b-b95d-c405f836db83","Type":"ContainerDied","Data":"9776260696fbea56b975c9482333a69947a388dd9cf6d3409362cf9f75b7fc83"} Oct 30 12:12:05.275831 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:05.270627 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-46f4f" event={"ID":"71215687-0701-4e7b-b95d-c405f836db83","Type":"ContainerDied","Data":"46cea526bdc7106c706608d5680c5934f402daa7e513ad3330a652de400417ee"} Oct 30 12:12:05.275831 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:05.270666 4468 scope.go:122] "RemoveContainer" containerID="9776260696fbea56b975c9482333a69947a388dd9cf6d3409362cf9f75b7fc83" Oct 30 12:12:05.275831 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:05.270882 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-46f4f" Oct 30 12:12:05.284446 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:05.284284 4468 scope.go:122] "RemoveContainer" containerID="9776260696fbea56b975c9482333a69947a388dd9cf6d3409362cf9f75b7fc83" Oct 30 12:12:05.284620 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:12:05.284596 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9776260696fbea56b975c9482333a69947a388dd9cf6d3409362cf9f75b7fc83\": container with ID starting with 9776260696fbea56b975c9482333a69947a388dd9cf6d3409362cf9f75b7fc83 not found: ID does not exist" containerID="9776260696fbea56b975c9482333a69947a388dd9cf6d3409362cf9f75b7fc83" Oct 30 12:12:05.284665 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:05.284631 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9776260696fbea56b975c9482333a69947a388dd9cf6d3409362cf9f75b7fc83"} err="failed to get container status \"9776260696fbea56b975c9482333a69947a388dd9cf6d3409362cf9f75b7fc83\": rpc error: code = NotFound desc = could not find container \"9776260696fbea56b975c9482333a69947a388dd9cf6d3409362cf9f75b7fc83\": container with ID starting with 9776260696fbea56b975c9482333a69947a388dd9cf6d3409362cf9f75b7fc83 not found: ID does not exist" Oct 30 12:12:05.321183 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:05.321142 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-46f4f"] Oct 30 12:12:05.333007 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:05.332976 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-46f4f"] Oct 30 12:12:07.089225 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:07.089190 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71215687-0701-4e7b-b95d-c405f836db83" path="/var/lib/kubelet/pods/71215687-0701-4e7b-b95d-c405f836db83/volumes" Oct 30 12:12:13.818040 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:13.818010 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v"] Oct 30 12:12:13.818459 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:13.818390 4468 cpu_manager.go:414] "containerMap: removing container" podUID="71215687-0701-4e7b-b95d-c405f836db83" containerName="manager" Oct 30 12:12:13.818459 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:13.818401 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71215687-0701-4e7b-b95d-c405f836db83" containerName="manager" Oct 30 12:12:13.818540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:13.818461 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="71215687-0701-4e7b-b95d-c405f836db83" containerName="manager" Oct 30 12:12:13.822193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:13.822168 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:13.837140 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:13.837099 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 12:12:13.848706 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:13.848672 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tqhfc\"" Oct 30 12:12:13.848706 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:13.848690 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 12:12:13.848706 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:13.848688 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 12:12:13.850434 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:13.850414 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v"] Oct 30 12:12:13.972219 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:13.972177 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede14942-b51f-4e10-94cc-153445945351-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-st46v\" (UID: \"ede14942-b51f-4e10-94cc-153445945351\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:13.972423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:13.972229 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede14942-b51f-4e10-94cc-153445945351-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-st46v\" (UID: \"ede14942-b51f-4e10-94cc-153445945351\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:13.972423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:13.972324 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede14942-b51f-4e10-94cc-153445945351-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-st46v\" (UID: \"ede14942-b51f-4e10-94cc-153445945351\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:13.972423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:13.972368 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede14942-b51f-4e10-94cc-153445945351-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-st46v\" (UID: \"ede14942-b51f-4e10-94cc-153445945351\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:13.972549 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:13.972429 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5zlz\" (UniqueName: \"kubernetes.io/projected/ede14942-b51f-4e10-94cc-153445945351-kube-api-access-k5zlz\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-st46v\" (UID: \"ede14942-b51f-4e10-94cc-153445945351\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:13.972549 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:13.972525 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede14942-b51f-4e10-94cc-153445945351-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-st46v\" (UID: \"ede14942-b51f-4e10-94cc-153445945351\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:14.073578 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:14.073474 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede14942-b51f-4e10-94cc-153445945351-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-st46v\" (UID: \"ede14942-b51f-4e10-94cc-153445945351\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:14.073578 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:14.073527 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede14942-b51f-4e10-94cc-153445945351-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-st46v\" (UID: \"ede14942-b51f-4e10-94cc-153445945351\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:14.073578 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:14.073547 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5zlz\" (UniqueName: \"kubernetes.io/projected/ede14942-b51f-4e10-94cc-153445945351-kube-api-access-k5zlz\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-st46v\" (UID: \"ede14942-b51f-4e10-94cc-153445945351\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:14.073848 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:14.073638 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede14942-b51f-4e10-94cc-153445945351-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-st46v\" (UID: \"ede14942-b51f-4e10-94cc-153445945351\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:14.073848 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:14.073705 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede14942-b51f-4e10-94cc-153445945351-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-st46v\" (UID: \"ede14942-b51f-4e10-94cc-153445945351\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:14.073848 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:14.073742 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede14942-b51f-4e10-94cc-153445945351-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-st46v\" (UID: \"ede14942-b51f-4e10-94cc-153445945351\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:14.074011 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:14.073976 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede14942-b51f-4e10-94cc-153445945351-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-st46v\" (UID: \"ede14942-b51f-4e10-94cc-153445945351\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:14.074105 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:14.074039 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede14942-b51f-4e10-94cc-153445945351-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-st46v\" (UID: \"ede14942-b51f-4e10-94cc-153445945351\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:14.074175 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:14.074156 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede14942-b51f-4e10-94cc-153445945351-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-st46v\" (UID: \"ede14942-b51f-4e10-94cc-153445945351\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:14.076057 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:14.076022 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede14942-b51f-4e10-94cc-153445945351-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-st46v\" (UID: \"ede14942-b51f-4e10-94cc-153445945351\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:14.076208 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:14.076169 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede14942-b51f-4e10-94cc-153445945351-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-st46v\" (UID: \"ede14942-b51f-4e10-94cc-153445945351\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:14.097002 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:14.096972 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5zlz\" (UniqueName: \"kubernetes.io/projected/ede14942-b51f-4e10-94cc-153445945351-kube-api-access-k5zlz\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-st46v\" (UID: \"ede14942-b51f-4e10-94cc-153445945351\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:14.135820 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:14.135781 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:14.457842 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:14.457809 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v"] Oct 30 12:12:14.458628 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:12:14.458603 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede14942_b51f_4e10_94cc_153445945351.slice/crio-f919e0facb97b63c2082d363f439b4a18bde58432728d5a1bfce784e4aa51862 WatchSource:0}: Error finding container f919e0facb97b63c2082d363f439b4a18bde58432728d5a1bfce784e4aa51862: Status 404 returned error can't find the container with id f919e0facb97b63c2082d363f439b4a18bde58432728d5a1bfce784e4aa51862 Oct 30 12:12:15.321445 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:15.321354 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" event={"ID":"ede14942-b51f-4e10-94cc-153445945351","Type":"ContainerStarted","Data":"f919e0facb97b63c2082d363f439b4a18bde58432728d5a1bfce784e4aa51862"} Oct 30 12:12:16.327505 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:16.327406 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" event={"ID":"ede14942-b51f-4e10-94cc-153445945351","Type":"ContainerStarted","Data":"b2bd2344798c5c7939da9bc8fd3a99134ea78a5b547c344442208cbcd53c65af"} Oct 30 12:12:16.369935 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:16.369874 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" podStartSLOduration=1.779224482 podStartE2EDuration="3.369860126s" totalImagesPullingTime="1.590635644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:16.368155505 +0000 UTC m=+1087.402293901" watchObservedRunningTime="2026-10-30 12:12:16.369860126 +0000 UTC m=+1087.403998519" Oct 30 12:12:17.136140 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:17.136100 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:19.279259 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.279224 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6"] Oct 30 12:12:19.281841 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.281815 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.307505 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.307467 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 12:12:19.310322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.310287 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6"] Oct 30 12:12:19.355338 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.355282 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ebfa71-f8e5-4c79-bf68-1cee6c31e544-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5ltm6\" (UID: \"75ebfa71-f8e5-4c79-bf68-1cee6c31e544\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.355532 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.355348 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ebfa71-f8e5-4c79-bf68-1cee6c31e544-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5ltm6\" (UID: \"75ebfa71-f8e5-4c79-bf68-1cee6c31e544\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.355532 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.355428 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ebfa71-f8e5-4c79-bf68-1cee6c31e544-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5ltm6\" (UID: \"75ebfa71-f8e5-4c79-bf68-1cee6c31e544\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.355532 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.355470 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ebfa71-f8e5-4c79-bf68-1cee6c31e544-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5ltm6\" (UID: \"75ebfa71-f8e5-4c79-bf68-1cee6c31e544\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.355532 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.355514 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ebfa71-f8e5-4c79-bf68-1cee6c31e544-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5ltm6\" (UID: \"75ebfa71-f8e5-4c79-bf68-1cee6c31e544\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.355693 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.355638 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m28bj\" (UniqueName: \"kubernetes.io/projected/75ebfa71-f8e5-4c79-bf68-1cee6c31e544-kube-api-access-m28bj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5ltm6\" (UID: \"75ebfa71-f8e5-4c79-bf68-1cee6c31e544\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.456457 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.456410 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ebfa71-f8e5-4c79-bf68-1cee6c31e544-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5ltm6\" (UID: \"75ebfa71-f8e5-4c79-bf68-1cee6c31e544\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.456457 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.456460 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ebfa71-f8e5-4c79-bf68-1cee6c31e544-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5ltm6\" (UID: \"75ebfa71-f8e5-4c79-bf68-1cee6c31e544\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.456722 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.456500 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ebfa71-f8e5-4c79-bf68-1cee6c31e544-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5ltm6\" (UID: \"75ebfa71-f8e5-4c79-bf68-1cee6c31e544\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.456722 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.456530 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ebfa71-f8e5-4c79-bf68-1cee6c31e544-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5ltm6\" (UID: \"75ebfa71-f8e5-4c79-bf68-1cee6c31e544\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.456722 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.456547 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m28bj\" (UniqueName: \"kubernetes.io/projected/75ebfa71-f8e5-4c79-bf68-1cee6c31e544-kube-api-access-m28bj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5ltm6\" (UID: \"75ebfa71-f8e5-4c79-bf68-1cee6c31e544\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.456722 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.456633 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ebfa71-f8e5-4c79-bf68-1cee6c31e544-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5ltm6\" (UID: \"75ebfa71-f8e5-4c79-bf68-1cee6c31e544\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.456949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.456923 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ebfa71-f8e5-4c79-bf68-1cee6c31e544-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5ltm6\" (UID: \"75ebfa71-f8e5-4c79-bf68-1cee6c31e544\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.456949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.456939 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ebfa71-f8e5-4c79-bf68-1cee6c31e544-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5ltm6\" (UID: \"75ebfa71-f8e5-4c79-bf68-1cee6c31e544\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.457033 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.456989 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ebfa71-f8e5-4c79-bf68-1cee6c31e544-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5ltm6\" (UID: \"75ebfa71-f8e5-4c79-bf68-1cee6c31e544\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.458886 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.458856 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ebfa71-f8e5-4c79-bf68-1cee6c31e544-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5ltm6\" (UID: \"75ebfa71-f8e5-4c79-bf68-1cee6c31e544\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.459046 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.459028 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ebfa71-f8e5-4c79-bf68-1cee6c31e544-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5ltm6\" (UID: \"75ebfa71-f8e5-4c79-bf68-1cee6c31e544\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.484020 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.483981 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m28bj\" (UniqueName: \"kubernetes.io/projected/75ebfa71-f8e5-4c79-bf68-1cee6c31e544-kube-api-access-m28bj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5ltm6\" (UID: \"75ebfa71-f8e5-4c79-bf68-1cee6c31e544\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.596802 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.596710 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:19.896594 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:19.896564 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6"] Oct 30 12:12:19.897379 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:12:19.897353 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ebfa71_f8e5_4c79_bf68_1cee6c31e544.slice/crio-9798d0cc93082c8266c38e229cc65d4d9971f3edfaa90bbacf35c7d2fdbc23fe WatchSource:0}: Error finding container 9798d0cc93082c8266c38e229cc65d4d9971f3edfaa90bbacf35c7d2fdbc23fe: Status 404 returned error can't find the container with id 9798d0cc93082c8266c38e229cc65d4d9971f3edfaa90bbacf35c7d2fdbc23fe Oct 30 12:12:20.352687 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:20.352652 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" event={"ID":"75ebfa71-f8e5-4c79-bf68-1cee6c31e544","Type":"ContainerStarted","Data":"c32dfe173589688d3fa316dc5daf7e5e9f6a64d42f5658a23413e6d224acfedb"} Oct 30 12:12:20.353110 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:20.352706 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" event={"ID":"75ebfa71-f8e5-4c79-bf68-1cee6c31e544","Type":"ContainerStarted","Data":"9798d0cc93082c8266c38e229cc65d4d9971f3edfaa90bbacf35c7d2fdbc23fe"} Oct 30 12:12:20.394694 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:20.394633 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" podStartSLOduration=1.196093504 podStartE2EDuration="1.394614622s" totalImagesPullingTime="198.521118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:20.393059203 +0000 UTC m=+1091.427197599" watchObservedRunningTime="2026-10-30 12:12:20.394614622 +0000 UTC m=+1091.428753017" Oct 30 12:12:20.597730 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:20.597689 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:23.565835 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.564906 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67"] Oct 30 12:12:23.576793 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.576763 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:23.605576 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.605544 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 12:12:23.609186 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.608967 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67"] Oct 30 12:12:23.618724 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.618685 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58wtq\" (UniqueName: \"kubernetes.io/projected/6abf7dd7-2cd8-463a-95a6-1c5b3d391f73-kube-api-access-58wtq\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2gm67\" (UID: \"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:23.619044 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.619011 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6abf7dd7-2cd8-463a-95a6-1c5b3d391f73-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2gm67\" (UID: \"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:23.619265 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.619244 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6abf7dd7-2cd8-463a-95a6-1c5b3d391f73-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2gm67\" (UID: \"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:23.619442 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.619412 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6abf7dd7-2cd8-463a-95a6-1c5b3d391f73-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2gm67\" (UID: \"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:23.619669 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.619650 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6abf7dd7-2cd8-463a-95a6-1c5b3d391f73-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2gm67\" (UID: \"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:23.619832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.619807 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6abf7dd7-2cd8-463a-95a6-1c5b3d391f73-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2gm67\" (UID: \"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:23.686154 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:12:23.679586 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:12:23.722604 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.722557 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6abf7dd7-2cd8-463a-95a6-1c5b3d391f73-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2gm67\" (UID: \"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:23.722914 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.722891 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6abf7dd7-2cd8-463a-95a6-1c5b3d391f73-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2gm67\" (UID: \"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:23.723123 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.723104 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6abf7dd7-2cd8-463a-95a6-1c5b3d391f73-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2gm67\" (UID: \"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:23.728093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.723257 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6abf7dd7-2cd8-463a-95a6-1c5b3d391f73-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2gm67\" (UID: \"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:23.728349 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.728325 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58wtq\" (UniqueName: \"kubernetes.io/projected/6abf7dd7-2cd8-463a-95a6-1c5b3d391f73-kube-api-access-58wtq\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2gm67\" (UID: \"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:23.728498 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.728479 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6abf7dd7-2cd8-463a-95a6-1c5b3d391f73-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2gm67\" (UID: \"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:23.737094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.729021 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6abf7dd7-2cd8-463a-95a6-1c5b3d391f73-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2gm67\" (UID: \"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:23.737365 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.727539 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6abf7dd7-2cd8-463a-95a6-1c5b3d391f73-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2gm67\" (UID: \"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:23.744549 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.744507 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6abf7dd7-2cd8-463a-95a6-1c5b3d391f73-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2gm67\" (UID: \"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:23.756867 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.756823 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6abf7dd7-2cd8-463a-95a6-1c5b3d391f73-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2gm67\" (UID: \"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:23.757533 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.757504 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6abf7dd7-2cd8-463a-95a6-1c5b3d391f73-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2gm67\" (UID: \"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:23.780130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.780096 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58wtq\" (UniqueName: \"kubernetes.io/projected/6abf7dd7-2cd8-463a-95a6-1c5b3d391f73-kube-api-access-58wtq\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2gm67\" (UID: \"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:23.912411 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:23.912289 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:24.138238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:24.138207 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:24.176152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:24.168441 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:24.499774 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:24.499365 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-st46v" Oct 30 12:12:24.530094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:24.529236 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67"] Oct 30 12:12:25.455045 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:25.455006 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" event={"ID":"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73","Type":"ContainerStarted","Data":"cd0208a8d9bd960ee66bd7744f28c7a16d191d228037c2555bd7fec20d554767"} Oct 30 12:12:25.455582 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:25.455082 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" event={"ID":"6abf7dd7-2cd8-463a-95a6-1c5b3d391f73","Type":"ContainerStarted","Data":"e7db4c8cc3931e8957e93da7b2cf1f4d466f73263aad4b1c37229c620165ea0d"} Oct 30 12:12:25.499170 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:25.499113 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" podStartSLOduration=1.710501541 podStartE2EDuration="2.499097796s" totalImagesPullingTime="788.596255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:25.497266448 +0000 UTC m=+1096.531404843" watchObservedRunningTime="2026-10-30 12:12:25.499097796 +0000 UTC m=+1096.533236183" Oct 30 12:12:25.912887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:25.912791 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:29.597640 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:29.597531 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:29.617169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:29.617137 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:30.537457 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:30.537421 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5ltm6" Oct 30 12:12:32.289697 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.289399 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z"] Oct 30 12:12:32.292191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.292167 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.313744 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.313711 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z"] Oct 30 12:12:32.320308 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.320279 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 12:12:32.333254 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.333211 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88k57\" (UniqueName: \"kubernetes.io/projected/f48b6092-fcf0-4787-adef-dba14c5df40f-kube-api-access-88k57\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z\" (UID: \"f48b6092-fcf0-4787-adef-dba14c5df40f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.333467 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.333336 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f48b6092-fcf0-4787-adef-dba14c5df40f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z\" (UID: \"f48b6092-fcf0-4787-adef-dba14c5df40f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.333467 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.333380 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f48b6092-fcf0-4787-adef-dba14c5df40f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z\" (UID: \"f48b6092-fcf0-4787-adef-dba14c5df40f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.333467 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.333427 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f48b6092-fcf0-4787-adef-dba14c5df40f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z\" (UID: \"f48b6092-fcf0-4787-adef-dba14c5df40f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.333637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.333586 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f48b6092-fcf0-4787-adef-dba14c5df40f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z\" (UID: \"f48b6092-fcf0-4787-adef-dba14c5df40f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.333691 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.333640 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f48b6092-fcf0-4787-adef-dba14c5df40f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z\" (UID: \"f48b6092-fcf0-4787-adef-dba14c5df40f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.434712 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.434670 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88k57\" (UniqueName: \"kubernetes.io/projected/f48b6092-fcf0-4787-adef-dba14c5df40f-kube-api-access-88k57\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z\" (UID: \"f48b6092-fcf0-4787-adef-dba14c5df40f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.434712 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.434717 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f48b6092-fcf0-4787-adef-dba14c5df40f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z\" (UID: \"f48b6092-fcf0-4787-adef-dba14c5df40f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.434947 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.434736 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f48b6092-fcf0-4787-adef-dba14c5df40f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z\" (UID: \"f48b6092-fcf0-4787-adef-dba14c5df40f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.434947 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.434849 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f48b6092-fcf0-4787-adef-dba14c5df40f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z\" (UID: \"f48b6092-fcf0-4787-adef-dba14c5df40f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.434947 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.434926 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f48b6092-fcf0-4787-adef-dba14c5df40f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z\" (UID: \"f48b6092-fcf0-4787-adef-dba14c5df40f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.435217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.434954 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f48b6092-fcf0-4787-adef-dba14c5df40f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z\" (UID: \"f48b6092-fcf0-4787-adef-dba14c5df40f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.435217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.435206 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f48b6092-fcf0-4787-adef-dba14c5df40f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z\" (UID: \"f48b6092-fcf0-4787-adef-dba14c5df40f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.435361 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.435336 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f48b6092-fcf0-4787-adef-dba14c5df40f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z\" (UID: \"f48b6092-fcf0-4787-adef-dba14c5df40f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.435410 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.435362 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f48b6092-fcf0-4787-adef-dba14c5df40f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z\" (UID: \"f48b6092-fcf0-4787-adef-dba14c5df40f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.437142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.437117 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f48b6092-fcf0-4787-adef-dba14c5df40f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z\" (UID: \"f48b6092-fcf0-4787-adef-dba14c5df40f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.437293 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.437277 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f48b6092-fcf0-4787-adef-dba14c5df40f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z\" (UID: \"f48b6092-fcf0-4787-adef-dba14c5df40f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.457356 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.457319 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88k57\" (UniqueName: \"kubernetes.io/projected/f48b6092-fcf0-4787-adef-dba14c5df40f-kube-api-access-88k57\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z\" (UID: \"f48b6092-fcf0-4787-adef-dba14c5df40f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.604320 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.604222 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:32.901397 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:32.901368 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z"] Oct 30 12:12:32.902578 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:12:32.902551 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf48b6092_fcf0_4787_adef_dba14c5df40f.slice/crio-b09d8cfc0aa510f892b4a1dc30259cd4f0bcde27b626d0340d4884e247366e49 WatchSource:0}: Error finding container b09d8cfc0aa510f892b4a1dc30259cd4f0bcde27b626d0340d4884e247366e49: Status 404 returned error can't find the container with id b09d8cfc0aa510f892b4a1dc30259cd4f0bcde27b626d0340d4884e247366e49 Oct 30 12:12:33.516566 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:33.516526 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" event={"ID":"f48b6092-fcf0-4787-adef-dba14c5df40f","Type":"ContainerStarted","Data":"afb988c56e8078f07706df24caaa637398e55d0ffc37a8664916801151edbd11"} Oct 30 12:12:33.517108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:33.516576 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" event={"ID":"f48b6092-fcf0-4787-adef-dba14c5df40f","Type":"ContainerStarted","Data":"b09d8cfc0aa510f892b4a1dc30259cd4f0bcde27b626d0340d4884e247366e49"} Oct 30 12:12:33.573984 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:33.573928 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podStartSLOduration=1.372978902 podStartE2EDuration="1.573912819s" totalImagesPullingTime="200.933917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:33.572010943 +0000 UTC m=+1104.606149341" watchObservedRunningTime="2026-10-30 12:12:33.573912819 +0000 UTC m=+1104.608051214" Oct 30 12:12:33.604890 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:33.604849 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:33.916093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:33.915560 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:33.935313 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:33.935270 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:34.549284 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:34.549028 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2gm67" Oct 30 12:12:42.604519 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:42.604476 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:42.617256 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:42.617222 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:43.596685 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:43.596655 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:12:48.445771 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:48.445738 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5949f4fc4c-bpf6c"] Oct 30 12:12:48.452516 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:48.452485 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5949f4fc4c-bpf6c" Oct 30 12:12:48.467189 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:48.467156 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5949f4fc4c-bpf6c"] Oct 30 12:12:48.523034 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:48.522994 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d4dab63-54cf-49ea-98a6-cfbf3adc9b68-tls-cert\") pod \"authorino-5949f4fc4c-bpf6c\" (UID: \"0d4dab63-54cf-49ea-98a6-cfbf3adc9b68\") " pod="kuadrant-system/authorino-5949f4fc4c-bpf6c" Oct 30 12:12:48.523225 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:48.523104 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sz5k\" (UniqueName: \"kubernetes.io/projected/0d4dab63-54cf-49ea-98a6-cfbf3adc9b68-kube-api-access-4sz5k\") pod \"authorino-5949f4fc4c-bpf6c\" (UID: \"0d4dab63-54cf-49ea-98a6-cfbf3adc9b68\") " pod="kuadrant-system/authorino-5949f4fc4c-bpf6c" Oct 30 12:12:48.623608 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:48.623562 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d4dab63-54cf-49ea-98a6-cfbf3adc9b68-tls-cert\") pod \"authorino-5949f4fc4c-bpf6c\" (UID: \"0d4dab63-54cf-49ea-98a6-cfbf3adc9b68\") " pod="kuadrant-system/authorino-5949f4fc4c-bpf6c" Oct 30 12:12:48.623608 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:48.623614 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sz5k\" (UniqueName: \"kubernetes.io/projected/0d4dab63-54cf-49ea-98a6-cfbf3adc9b68-kube-api-access-4sz5k\") pod \"authorino-5949f4fc4c-bpf6c\" (UID: \"0d4dab63-54cf-49ea-98a6-cfbf3adc9b68\") " pod="kuadrant-system/authorino-5949f4fc4c-bpf6c" Oct 30 12:12:48.626306 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:48.626271 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d4dab63-54cf-49ea-98a6-cfbf3adc9b68-tls-cert\") pod \"authorino-5949f4fc4c-bpf6c\" (UID: \"0d4dab63-54cf-49ea-98a6-cfbf3adc9b68\") " pod="kuadrant-system/authorino-5949f4fc4c-bpf6c" Oct 30 12:12:48.646599 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:48.646567 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sz5k\" (UniqueName: \"kubernetes.io/projected/0d4dab63-54cf-49ea-98a6-cfbf3adc9b68-kube-api-access-4sz5k\") pod \"authorino-5949f4fc4c-bpf6c\" (UID: \"0d4dab63-54cf-49ea-98a6-cfbf3adc9b68\") " pod="kuadrant-system/authorino-5949f4fc4c-bpf6c" Oct 30 12:12:48.766618 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:48.766529 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5949f4fc4c-bpf6c" Oct 30 12:12:49.064004 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:49.063973 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5949f4fc4c-bpf6c"] Oct 30 12:12:49.065459 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:12:49.065420 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d4dab63_54cf_49ea_98a6_cfbf3adc9b68.slice/crio-a7c8668e78fbebb44380258882bbd4903057c4a39de7102565558a38e20c74b1 WatchSource:0}: Error finding container a7c8668e78fbebb44380258882bbd4903057c4a39de7102565558a38e20c74b1: Status 404 returned error can't find the container with id a7c8668e78fbebb44380258882bbd4903057c4a39de7102565558a38e20c74b1 Oct 30 12:12:49.614114 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:49.614061 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5949f4fc4c-bpf6c" event={"ID":"0d4dab63-54cf-49ea-98a6-cfbf3adc9b68","Type":"ContainerStarted","Data":"a7c8668e78fbebb44380258882bbd4903057c4a39de7102565558a38e20c74b1"} Oct 30 12:12:50.623228 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:50.623188 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5949f4fc4c-bpf6c" event={"ID":"0d4dab63-54cf-49ea-98a6-cfbf3adc9b68","Type":"ContainerStarted","Data":"b6b45bf1b4ef7bccfa08e2d2f9344990568855a46d02057155d4f45936e988b7"} Oct 30 12:12:50.661711 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:50.661651 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5949f4fc4c-bpf6c" podStartSLOduration=2.005601017 podStartE2EDuration="2.661636596s" totalImagesPullingTime="656.035579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:50.661447185 +0000 UTC m=+1121.695585585" watchObservedRunningTime="2026-10-30 12:12:50.661636596 +0000 UTC m=+1121.695774992" Oct 30 12:12:50.689081 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:50.689020 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f96c4569b-7p5fs"] Oct 30 12:12:50.689447 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:50.689418 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5f96c4569b-7p5fs" podUID="e3fdd246-9b07-47a5-8736-08de39cb55cd" containerName="authorino" containerID="cri-o://9d588cae9c8b7c852d0a89bda0bb41933a9a5775251de624913414b6e199b393" gracePeriod=30 Oct 30 12:12:50.951530 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:50.951502 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f96c4569b-7p5fs" Oct 30 12:12:51.048307 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:51.048262 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3fdd246-9b07-47a5-8736-08de39cb55cd-tls-cert\" (UniqueName: \"kubernetes.io/secret/e3fdd246-9b07-47a5-8736-08de39cb55cd-tls-cert\") pod \"e3fdd246-9b07-47a5-8736-08de39cb55cd\" (UID: \"e3fdd246-9b07-47a5-8736-08de39cb55cd\") " Oct 30 12:12:51.048507 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:51.048343 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3fdd246-9b07-47a5-8736-08de39cb55cd-kube-api-access-ns5v9\" (UniqueName: \"kubernetes.io/projected/e3fdd246-9b07-47a5-8736-08de39cb55cd-kube-api-access-ns5v9\") pod \"e3fdd246-9b07-47a5-8736-08de39cb55cd\" (UID: \"e3fdd246-9b07-47a5-8736-08de39cb55cd\") " Oct 30 12:12:51.050491 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:51.050453 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3fdd246-9b07-47a5-8736-08de39cb55cd-kube-api-access-ns5v9" pod "e3fdd246-9b07-47a5-8736-08de39cb55cd" (UID: "e3fdd246-9b07-47a5-8736-08de39cb55cd"). InnerVolumeSpecName "kube-api-access-ns5v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:12:51.060080 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:51.060036 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3fdd246-9b07-47a5-8736-08de39cb55cd-tls-cert" pod "e3fdd246-9b07-47a5-8736-08de39cb55cd" (UID: "e3fdd246-9b07-47a5-8736-08de39cb55cd"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:12:51.150043 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:51.150008 4468 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e3fdd246-9b07-47a5-8736-08de39cb55cd-tls-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:12:51.150043 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:51.150042 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns5v9\" (UniqueName: \"kubernetes.io/projected/e3fdd246-9b07-47a5-8736-08de39cb55cd-kube-api-access-ns5v9\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:12:51.636091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:51.635633 4468 generic.go:356] "Generic (PLEG): container finished" podID="e3fdd246-9b07-47a5-8736-08de39cb55cd" containerID="9d588cae9c8b7c852d0a89bda0bb41933a9a5775251de624913414b6e199b393" exitCode=0 Oct 30 12:12:51.640155 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:51.636748 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f96c4569b-7p5fs" Oct 30 12:12:51.640319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:51.640205 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f96c4569b-7p5fs" event={"ID":"e3fdd246-9b07-47a5-8736-08de39cb55cd","Type":"ContainerDied","Data":"9d588cae9c8b7c852d0a89bda0bb41933a9a5775251de624913414b6e199b393"} Oct 30 12:12:51.640319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:51.640252 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f96c4569b-7p5fs" event={"ID":"e3fdd246-9b07-47a5-8736-08de39cb55cd","Type":"ContainerDied","Data":"1d07a4664f07b2e061fbe56a5d50714f353d6d42154d92c19632b70a0e5754f4"} Oct 30 12:12:51.640319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:51.640272 4468 scope.go:122] "RemoveContainer" containerID="9d588cae9c8b7c852d0a89bda0bb41933a9a5775251de624913414b6e199b393" Oct 30 12:12:51.653726 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:51.653704 4468 scope.go:122] "RemoveContainer" containerID="9d588cae9c8b7c852d0a89bda0bb41933a9a5775251de624913414b6e199b393" Oct 30 12:12:51.654124 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:12:51.654096 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d588cae9c8b7c852d0a89bda0bb41933a9a5775251de624913414b6e199b393\": container with ID starting with 9d588cae9c8b7c852d0a89bda0bb41933a9a5775251de624913414b6e199b393 not found: ID does not exist" containerID="9d588cae9c8b7c852d0a89bda0bb41933a9a5775251de624913414b6e199b393" Oct 30 12:12:51.654233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:51.654138 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d588cae9c8b7c852d0a89bda0bb41933a9a5775251de624913414b6e199b393"} err="failed to get container status \"9d588cae9c8b7c852d0a89bda0bb41933a9a5775251de624913414b6e199b393\": rpc error: code = NotFound desc = could not find container \"9d588cae9c8b7c852d0a89bda0bb41933a9a5775251de624913414b6e199b393\": container with ID starting with 9d588cae9c8b7c852d0a89bda0bb41933a9a5775251de624913414b6e199b393 not found: ID does not exist" Oct 30 12:12:51.694332 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:51.694296 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f96c4569b-7p5fs"] Oct 30 12:12:51.706509 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:51.706469 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f96c4569b-7p5fs"] Oct 30 12:12:53.088255 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:12:53.088222 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3fdd246-9b07-47a5-8736-08de39cb55cd" path="/var/lib/kubelet/pods/e3fdd246-9b07-47a5-8736-08de39cb55cd/volumes" Oct 30 12:12:53.600509 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:12:53.600469 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:13:21.991426 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:21.991384 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn"] Oct 30 12:13:21.992057 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:21.992040 4468 cpu_manager.go:414] "containerMap: removing container" podUID="e3fdd246-9b07-47a5-8736-08de39cb55cd" containerName="authorino" Oct 30 12:13:21.992162 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:21.992059 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3fdd246-9b07-47a5-8736-08de39cb55cd" containerName="authorino" Oct 30 12:13:21.992219 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:21.992208 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3fdd246-9b07-47a5-8736-08de39cb55cd" containerName="authorino" Oct 30 12:13:21.996835 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:21.996808 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.026268 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.026230 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-80143318-openshift-default-dockercfg-448hc\"" Oct 30 12:13:22.045559 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.045510 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn"] Oct 30 12:13:22.075629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.075592 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e74d16-3dde-444b-8f10-769314d02026-istiod-ca-cert\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.075810 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.075641 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-workload-socket\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.075810 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.075727 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e74d16-3dde-444b-8f10-769314d02026-istio-podinfo\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.075810 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.075750 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-istio-envoy\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.075810 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.075786 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l5f8\" (UniqueName: \"kubernetes.io/projected/d5e74d16-3dde-444b-8f10-769314d02026-kube-api-access-7l5f8\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.075945 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.075854 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-workload-certs\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.075945 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.075886 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5e74d16-3dde-444b-8f10-769314d02026-istio-token\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.075945 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.075922 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-credential-socket\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.076035 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.075963 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-istio-data\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.176703 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.176661 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e74d16-3dde-444b-8f10-769314d02026-istiod-ca-cert\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.176865 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.176717 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-workload-socket\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.176865 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.176764 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e74d16-3dde-444b-8f10-769314d02026-istio-podinfo\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.176865 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.176793 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-istio-envoy\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.176865 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.176835 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l5f8\" (UniqueName: \"kubernetes.io/projected/d5e74d16-3dde-444b-8f10-769314d02026-kube-api-access-7l5f8\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.177150 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.176894 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-workload-certs\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.177150 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.176933 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5e74d16-3dde-444b-8f10-769314d02026-istio-token\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.177150 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.176984 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-credential-socket\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.177150 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.177016 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-istio-data\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.177367 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.177240 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-workload-socket\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.177367 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.177342 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-workload-certs\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.177461 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.177418 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-istio-data\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.177517 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.177501 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-credential-socket\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.177603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.177584 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e74d16-3dde-444b-8f10-769314d02026-istiod-ca-cert\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.179261 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.179234 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-istio-envoy\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.179443 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.179425 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e74d16-3dde-444b-8f10-769314d02026-istio-podinfo\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.205308 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.205239 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l5f8\" (UniqueName: \"kubernetes.io/projected/d5e74d16-3dde-444b-8f10-769314d02026-kube-api-access-7l5f8\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.206849 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.206317 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv"] Oct 30 12:13:22.210940 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.210877 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5e74d16-3dde-444b-8f10-769314d02026-istio-token\") pod \"e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.211682 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.211658 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.230461 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.230433 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1c85bd66-openshift-default-dockercfg-4d6pc\"" Oct 30 12:13:22.239281 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.239251 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv"] Oct 30 12:13:22.278164 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.278042 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-token\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.278164 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.278143 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-podinfo\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.278413 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.278196 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-workload-socket\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.278413 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.278280 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istiod-ca-cert\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.278413 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.278317 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-credential-socket\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.278413 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.278384 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-workload-certs\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.278613 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.278443 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gppgm\" (UniqueName: \"kubernetes.io/projected/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-kube-api-access-gppgm\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.278613 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.278538 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-envoy\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.278613 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.278588 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-data\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.311906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.311873 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:22.396037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.380332 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-podinfo\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.396037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.380409 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-workload-socket\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.396037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.381831 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istiod-ca-cert\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.396037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.381814 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istiod-ca-cert\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.396037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.383363 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-credential-socket\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.396037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.382177 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-workload-socket\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.396037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.383417 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-workload-certs\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.396037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.383495 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gppgm\" (UniqueName: \"kubernetes.io/projected/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-kube-api-access-gppgm\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.396037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.383556 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-envoy\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.396037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.383611 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-data\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.396037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.383662 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-token\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.396037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.383829 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-credential-socket\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.396037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.384389 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-workload-certs\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.396037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.384687 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-data\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.396037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.387376 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-envoy\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.411994 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.411680 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-podinfo\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.421472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.421435 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-token\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.421647 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.421503 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gppgm\" (UniqueName: \"kubernetes.io/projected/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-kube-api-access-gppgm\") pod \"e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.546441 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.546343 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:22.636471 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.635933 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn"] Oct 30 12:13:22.637661 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:13:22.637179 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e74d16_3dde_444b_8f10_769314d02026.slice/crio-77faf8b6462ec60d6f1d332a852791f223fc1e10f91fd0961864e3708fc03092 WatchSource:0}: Error finding container 77faf8b6462ec60d6f1d332a852791f223fc1e10f91fd0961864e3708fc03092: Status 404 returned error can't find the container with id 77faf8b6462ec60d6f1d332a852791f223fc1e10f91fd0961864e3708fc03092 Oct 30 12:13:22.657003 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.640741 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:13:22.657003 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.640827 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:13:22.657003 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.640858 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:13:22.796223 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.796191 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" event={"ID":"d5e74d16-3dde-444b-8f10-769314d02026","Type":"ContainerStarted","Data":"0fc49bd91096e7afd1d01a24d6e748edc6b1419d2c8358803cb1ff7ca5458a01"} Oct 30 12:13:22.796404 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.796250 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" event={"ID":"d5e74d16-3dde-444b-8f10-769314d02026","Type":"ContainerStarted","Data":"77faf8b6462ec60d6f1d332a852791f223fc1e10f91fd0961864e3708fc03092"} Oct 30 12:13:22.857596 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.857485 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" podStartSLOduration=1.857466973 podStartE2EDuration="1.857466973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:22.85580177 +0000 UTC m=+1153.889940166" watchObservedRunningTime="2026-10-30 12:13:22.857466973 +0000 UTC m=+1153.891605379" Oct 30 12:13:22.903775 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.903744 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv"] Oct 30 12:13:22.905210 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:13:22.905182 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eef0ee0_246f_4a05_9ba0_8b0746f23c2b.slice/crio-6f835e2382055c950ca18a14cccbbc950aea0ee1db20d875568afc365921650e WatchSource:0}: Error finding container 6f835e2382055c950ca18a14cccbbc950aea0ee1db20d875568afc365921650e: Status 404 returned error can't find the container with id 6f835e2382055c950ca18a14cccbbc950aea0ee1db20d875568afc365921650e Oct 30 12:13:22.907538 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.907498 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:13:22.907611 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.907592 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:13:22.907665 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:22.907650 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:13:23.312675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:23.312638 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:23.317920 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:23.317889 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:23.599448 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:13:23.599353 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:13:23.803155 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:23.803106 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" event={"ID":"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b","Type":"ContainerStarted","Data":"544320268b1126839dc8d7b3f0e7037a30d6171d202513f4bd6842b2a280f3d3"} Oct 30 12:13:23.803357 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:23.803173 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" event={"ID":"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b","Type":"ContainerStarted","Data":"6f835e2382055c950ca18a14cccbbc950aea0ee1db20d875568afc365921650e"} Oct 30 12:13:23.803615 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:23.803556 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:23.804508 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:23.804485 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:23.851599 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:23.851468 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" podStartSLOduration=1.851453093 podStartE2EDuration="1.851453093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:23.849595345 +0000 UTC m=+1154.883733742" watchObservedRunningTime="2026-10-30 12:13:23.851453093 +0000 UTC m=+1154.885591488" Oct 30 12:13:24.546768 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:24.546727 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:24.553041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:24.553012 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:24.811499 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:24.811406 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:24.814925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:24.814896 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:26.226379 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.226336 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6"] Oct 30 12:13:26.230312 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.230284 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6" Oct 30 12:13:26.262788 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.262756 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-307283d2\"" Oct 30 12:13:26.262978 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.262930 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-307283d2-dockercfg-bwr4f\"" Oct 30 12:13:26.265182 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.265148 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6"] Oct 30 12:13:26.281501 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.281467 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6"] Oct 30 12:13:26.285490 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.285462 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6" Oct 30 12:13:26.313225 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.313193 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6"] Oct 30 12:13:26.328775 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.328740 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f581c0ea-a55a-4a4a-b892-45c8759ba074-plugins-config-volume\") pod \"payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6\" (UID: \"f581c0ea-a55a-4a4a-b892-45c8759ba074\") " pod="openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6" Oct 30 12:13:26.328982 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.328785 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv2j7\" (UniqueName: \"kubernetes.io/projected/dac85f8a-9ce0-449f-b7a3-277c53941fd0-kube-api-access-wv2j7\") pod \"payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6\" (UID: \"dac85f8a-9ce0-449f-b7a3-277c53941fd0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6" Oct 30 12:13:26.328982 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.328862 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x86x6\" (UniqueName: \"kubernetes.io/projected/f581c0ea-a55a-4a4a-b892-45c8759ba074-kube-api-access-x86x6\") pod \"payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6\" (UID: \"f581c0ea-a55a-4a4a-b892-45c8759ba074\") " pod="openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6" Oct 30 12:13:26.329117 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.329047 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac85f8a-9ce0-449f-b7a3-277c53941fd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6\" (UID: \"dac85f8a-9ce0-449f-b7a3-277c53941fd0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6" Oct 30 12:13:26.433730 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.433687 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f581c0ea-a55a-4a4a-b892-45c8759ba074-plugins-config-volume\") pod \"payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6\" (UID: \"f581c0ea-a55a-4a4a-b892-45c8759ba074\") " pod="openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6" Oct 30 12:13:26.435158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.435127 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2j7\" (UniqueName: \"kubernetes.io/projected/dac85f8a-9ce0-449f-b7a3-277c53941fd0-kube-api-access-wv2j7\") pod \"payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6\" (UID: \"dac85f8a-9ce0-449f-b7a3-277c53941fd0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6" Oct 30 12:13:26.435719 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.435688 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x86x6\" (UniqueName: \"kubernetes.io/projected/f581c0ea-a55a-4a4a-b892-45c8759ba074-kube-api-access-x86x6\") pod \"payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6\" (UID: \"f581c0ea-a55a-4a4a-b892-45c8759ba074\") " pod="openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6" Oct 30 12:13:26.444319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.444279 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac85f8a-9ce0-449f-b7a3-277c53941fd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6\" (UID: \"dac85f8a-9ce0-449f-b7a3-277c53941fd0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6" Oct 30 12:13:26.445734 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.435006 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f581c0ea-a55a-4a4a-b892-45c8759ba074-plugins-config-volume\") pod \"payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6\" (UID: \"f581c0ea-a55a-4a4a-b892-45c8759ba074\") " pod="openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6" Oct 30 12:13:26.459739 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.445456 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac85f8a-9ce0-449f-b7a3-277c53941fd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6\" (UID: \"dac85f8a-9ce0-449f-b7a3-277c53941fd0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6" Oct 30 12:13:26.472085 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.472030 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv2j7\" (UniqueName: \"kubernetes.io/projected/dac85f8a-9ce0-449f-b7a3-277c53941fd0-kube-api-access-wv2j7\") pod \"payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6\" (UID: \"dac85f8a-9ce0-449f-b7a3-277c53941fd0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6" Oct 30 12:13:26.472446 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.472342 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x86x6\" (UniqueName: \"kubernetes.io/projected/f581c0ea-a55a-4a4a-b892-45c8759ba074-kube-api-access-x86x6\") pod \"payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6\" (UID: \"f581c0ea-a55a-4a4a-b892-45c8759ba074\") " pod="openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6" Oct 30 12:13:26.543707 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.543620 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6" Oct 30 12:13:26.602372 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.602304 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6" Oct 30 12:13:26.854780 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.854746 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6"] Oct 30 12:13:26.854960 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:13:26.854939 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac85f8a_9ce0_449f_b7a3_277c53941fd0.slice/crio-a78358f9cd5c48c16288df39c8b906ac0913341c4d9d7a263920ad33867f060d WatchSource:0}: Error finding container a78358f9cd5c48c16288df39c8b906ac0913341c4d9d7a263920ad33867f060d: Status 404 returned error can't find the container with id a78358f9cd5c48c16288df39c8b906ac0913341c4d9d7a263920ad33867f060d Oct 30 12:13:26.928483 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:26.928440 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6"] Oct 30 12:13:26.929910 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:13:26.929878 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf581c0ea_a55a_4a4a_b892_45c8759ba074.slice/crio-fcbe24466bf84b2ba965a1c6e24b958fc2928d1aa67f4eafe6952bdace6097e2 WatchSource:0}: Error finding container fcbe24466bf84b2ba965a1c6e24b958fc2928d1aa67f4eafe6952bdace6097e2: Status 404 returned error can't find the container with id fcbe24466bf84b2ba965a1c6e24b958fc2928d1aa67f4eafe6952bdace6097e2 Oct 30 12:13:27.828577 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.828539 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6" event={"ID":"dac85f8a-9ce0-449f-b7a3-277c53941fd0","Type":"ContainerStarted","Data":"782a0b7cba571be37b744350916cae871058901c399c0cecdc542bc91475393c"} Oct 30 12:13:27.829125 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.828604 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6" event={"ID":"dac85f8a-9ce0-449f-b7a3-277c53941fd0","Type":"ContainerStarted","Data":"a78358f9cd5c48c16288df39c8b906ac0913341c4d9d7a263920ad33867f060d"} Oct 30 12:13:27.829125 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.828635 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6" Oct 30 12:13:27.830426 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.830384 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6" event={"ID":"f581c0ea-a55a-4a4a-b892-45c8759ba074","Type":"ContainerStarted","Data":"a8c0ae2173bff9b55a7a668059f10ad97b888e8f1895a3ea63f06bba636e76ce"} Oct 30 12:13:27.830598 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.830438 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6" event={"ID":"f581c0ea-a55a-4a4a-b892-45c8759ba074","Type":"ContainerStarted","Data":"fcbe24466bf84b2ba965a1c6e24b958fc2928d1aa67f4eafe6952bdace6097e2"} Oct 30 12:13:27.830956 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.830938 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6" Oct 30 12:13:27.834627 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.834605 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t"] Oct 30 12:13:27.838711 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.838691 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t" Oct 30 12:13:27.856323 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.856295 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t"] Oct 30 12:13:27.860515 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.860479 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkjt6\" (UniqueName: \"kubernetes.io/projected/1be0127f-6554-4c65-be90-db22d98a6071-kube-api-access-bkjt6\") pod \"payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t\" (UID: \"1be0127f-6554-4c65-be90-db22d98a6071\") " pod="openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t" Oct 30 12:13:27.860699 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.860554 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be0127f-6554-4c65-be90-db22d98a6071-plugins-config-volume\") pod \"payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t\" (UID: \"1be0127f-6554-4c65-be90-db22d98a6071\") " pod="openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t" Oct 30 12:13:27.901368 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.901312 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6" podStartSLOduration=1.901298541 podStartE2EDuration="1.901298541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:27.871951435 +0000 UTC m=+1158.906089830" watchObservedRunningTime="2026-10-30 12:13:27.901298541 +0000 UTC m=+1158.935436936" Oct 30 12:13:27.901809 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.901795 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p"] Oct 30 12:13:27.905384 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.905366 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p" Oct 30 12:13:27.917737 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.917692 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6" podStartSLOduration=1.917679563 podStartE2EDuration="1.917679563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:27.917295137 +0000 UTC m=+1158.951433533" watchObservedRunningTime="2026-10-30 12:13:27.917679563 +0000 UTC m=+1158.951817958" Oct 30 12:13:27.924784 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.924746 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-80143318-dockercfg-gmbtv\"" Oct 30 12:13:27.924946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.924787 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p"] Oct 30 12:13:27.932369 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.932341 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-80143318\"" Oct 30 12:13:27.961853 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.961821 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkjt6\" (UniqueName: \"kubernetes.io/projected/1be0127f-6554-4c65-be90-db22d98a6071-kube-api-access-bkjt6\") pod \"payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t\" (UID: \"1be0127f-6554-4c65-be90-db22d98a6071\") " pod="openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t" Oct 30 12:13:27.962086 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.961875 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be0127f-6554-4c65-be90-db22d98a6071-plugins-config-volume\") pod \"payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t\" (UID: \"1be0127f-6554-4c65-be90-db22d98a6071\") " pod="openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t" Oct 30 12:13:27.962086 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.961915 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184b5d2e-de0b-4f06-bbb5-37522055a6e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p\" (UID: \"184b5d2e-de0b-4f06-bbb5-37522055a6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p" Oct 30 12:13:27.962086 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.961955 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d4gl\" (UniqueName: \"kubernetes.io/projected/184b5d2e-de0b-4f06-bbb5-37522055a6e8-kube-api-access-4d4gl\") pod \"payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p\" (UID: \"184b5d2e-de0b-4f06-bbb5-37522055a6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p" Oct 30 12:13:27.962899 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.962873 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be0127f-6554-4c65-be90-db22d98a6071-plugins-config-volume\") pod \"payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t\" (UID: \"1be0127f-6554-4c65-be90-db22d98a6071\") " pod="openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t" Oct 30 12:13:27.985097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:27.985033 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkjt6\" (UniqueName: \"kubernetes.io/projected/1be0127f-6554-4c65-be90-db22d98a6071-kube-api-access-bkjt6\") pod \"payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t\" (UID: \"1be0127f-6554-4c65-be90-db22d98a6071\") " pod="openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t" Oct 30 12:13:28.065675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:28.065632 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184b5d2e-de0b-4f06-bbb5-37522055a6e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p\" (UID: \"184b5d2e-de0b-4f06-bbb5-37522055a6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p" Oct 30 12:13:28.067004 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:28.066868 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184b5d2e-de0b-4f06-bbb5-37522055a6e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p\" (UID: \"184b5d2e-de0b-4f06-bbb5-37522055a6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p" Oct 30 12:13:28.067004 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:28.066974 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d4gl\" (UniqueName: \"kubernetes.io/projected/184b5d2e-de0b-4f06-bbb5-37522055a6e8-kube-api-access-4d4gl\") pod \"payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p\" (UID: \"184b5d2e-de0b-4f06-bbb5-37522055a6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p" Oct 30 12:13:28.099923 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:28.099834 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d4gl\" (UniqueName: \"kubernetes.io/projected/184b5d2e-de0b-4f06-bbb5-37522055a6e8-kube-api-access-4d4gl\") pod \"payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p\" (UID: \"184b5d2e-de0b-4f06-bbb5-37522055a6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p" Oct 30 12:13:28.156361 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:28.156314 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t" Oct 30 12:13:28.222242 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:28.220413 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p" Oct 30 12:13:28.457719 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:28.457686 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t"] Oct 30 12:13:28.458358 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:13:28.458330 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be0127f_6554_4c65_be90_db22d98a6071.slice/crio-65c10e1600bac01b20d7bbabb06a6c133a07127199a22d9af287fd8278341c5f WatchSource:0}: Error finding container 65c10e1600bac01b20d7bbabb06a6c133a07127199a22d9af287fd8278341c5f: Status 404 returned error can't find the container with id 65c10e1600bac01b20d7bbabb06a6c133a07127199a22d9af287fd8278341c5f Oct 30 12:13:28.530375 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:13:28.527508 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184b5d2e_de0b_4f06_bbb5_37522055a6e8.slice/crio-2ca3e4f48110ddca087fa10cf0bbd2ea9120e3e2bec2ddef2e246ba82f39e2e4 WatchSource:0}: Error finding container 2ca3e4f48110ddca087fa10cf0bbd2ea9120e3e2bec2ddef2e246ba82f39e2e4: Status 404 returned error can't find the container with id 2ca3e4f48110ddca087fa10cf0bbd2ea9120e3e2bec2ddef2e246ba82f39e2e4 Oct 30 12:13:28.531567 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:28.531493 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p"] Oct 30 12:13:28.837279 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:28.837179 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t" event={"ID":"1be0127f-6554-4c65-be90-db22d98a6071","Type":"ContainerStarted","Data":"039b2dcfd1e42bc947771ea02827bc507c472288f973ff2877de7ba2de5cf018"} Oct 30 12:13:28.837279 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:28.837238 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t" event={"ID":"1be0127f-6554-4c65-be90-db22d98a6071","Type":"ContainerStarted","Data":"65c10e1600bac01b20d7bbabb06a6c133a07127199a22d9af287fd8278341c5f"} Oct 30 12:13:28.837279 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:28.837268 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t" Oct 30 12:13:28.838801 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:28.838767 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p" event={"ID":"184b5d2e-de0b-4f06-bbb5-37522055a6e8","Type":"ContainerStarted","Data":"b127c61e210cc540d32191a6d5383eb78aea753f7ca44abd3335c5d0f3488a2b"} Oct 30 12:13:28.838911 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:28.838814 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p" event={"ID":"184b5d2e-de0b-4f06-bbb5-37522055a6e8","Type":"ContainerStarted","Data":"2ca3e4f48110ddca087fa10cf0bbd2ea9120e3e2bec2ddef2e246ba82f39e2e4"} Oct 30 12:13:28.838911 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:28.838865 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p" Oct 30 12:13:28.876922 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:28.876861 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t" podStartSLOduration=1.8768427939999999 podStartE2EDuration="1.876842794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:28.875541816 +0000 UTC m=+1159.909680212" watchObservedRunningTime="2026-10-30 12:13:28.876842794 +0000 UTC m=+1159.910981190" Oct 30 12:13:28.915481 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:28.915425 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p" podStartSLOduration=1.9154085969999999 podStartE2EDuration="1.915408597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:28.914413246 +0000 UTC m=+1159.948551643" watchObservedRunningTime="2026-10-30 12:13:28.915408597 +0000 UTC m=+1159.949546992" Oct 30 12:13:29.451663 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.451624 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7"] Oct 30 12:13:29.455628 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.455603 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7" Oct 30 12:13:29.474925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.474891 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7"] Oct 30 12:13:29.476299 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.476274 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1c85bd66-dockercfg-t7nsj\"" Oct 30 12:13:29.492856 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.492821 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb-plugins-config-volume\") pod \"payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7\" (UID: \"20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb\") " pod="openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7" Oct 30 12:13:29.493050 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.492870 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njf8v\" (UniqueName: \"kubernetes.io/projected/20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb-kube-api-access-njf8v\") pod \"payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7\" (UID: \"20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb\") " pod="openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7" Oct 30 12:13:29.516790 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.516752 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb"] Oct 30 12:13:29.520984 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.520957 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb" Oct 30 12:13:29.527673 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.527648 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1c85bd66\"" Oct 30 12:13:29.539968 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.539924 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb"] Oct 30 12:13:29.609811 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.609773 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb-plugins-config-volume\") pod \"payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7\" (UID: \"20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb\") " pod="openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7" Oct 30 12:13:29.617685 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.616231 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njf8v\" (UniqueName: \"kubernetes.io/projected/20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb-kube-api-access-njf8v\") pod \"payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7\" (UID: \"20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb\") " pod="openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7" Oct 30 12:13:29.617685 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.616322 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0ce46a-bfff-4509-97ae-58fca797cc56-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb\" (UID: \"cc0ce46a-bfff-4509-97ae-58fca797cc56\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb" Oct 30 12:13:29.617685 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.616395 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xjw4\" (UniqueName: \"kubernetes.io/projected/cc0ce46a-bfff-4509-97ae-58fca797cc56-kube-api-access-7xjw4\") pod \"payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb\" (UID: \"cc0ce46a-bfff-4509-97ae-58fca797cc56\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb" Oct 30 12:13:29.617685 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.610971 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb-plugins-config-volume\") pod \"payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7\" (UID: \"20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb\") " pod="openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7" Oct 30 12:13:29.641268 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.640596 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njf8v\" (UniqueName: \"kubernetes.io/projected/20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb-kube-api-access-njf8v\") pod \"payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7\" (UID: \"20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb\") " pod="openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7" Oct 30 12:13:29.717766 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.717670 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0ce46a-bfff-4509-97ae-58fca797cc56-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb\" (UID: \"cc0ce46a-bfff-4509-97ae-58fca797cc56\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb" Oct 30 12:13:29.717766 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.717729 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xjw4\" (UniqueName: \"kubernetes.io/projected/cc0ce46a-bfff-4509-97ae-58fca797cc56-kube-api-access-7xjw4\") pod \"payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb\" (UID: \"cc0ce46a-bfff-4509-97ae-58fca797cc56\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb" Oct 30 12:13:29.718476 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.718454 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0ce46a-bfff-4509-97ae-58fca797cc56-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb\" (UID: \"cc0ce46a-bfff-4509-97ae-58fca797cc56\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb" Oct 30 12:13:29.751276 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.751242 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xjw4\" (UniqueName: \"kubernetes.io/projected/cc0ce46a-bfff-4509-97ae-58fca797cc56-kube-api-access-7xjw4\") pod \"payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb\" (UID: \"cc0ce46a-bfff-4509-97ae-58fca797cc56\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb" Oct 30 12:13:29.769438 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.769402 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7" Oct 30 12:13:29.843192 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:29.842878 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb" Oct 30 12:13:30.081254 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.081163 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h"] Oct 30 12:13:30.090046 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.086789 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.090046 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.086684 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7"] Oct 30 12:13:30.090302 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:13:30.090148 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20aa3ee4_4dc0_4c81_93df_04db6ceaa7fb.slice/crio-23a322eb3a4ecfdfe6a83e59d5e54f0716990307a7a2830a18632b40ca566943 WatchSource:0}: Error finding container 23a322eb3a4ecfdfe6a83e59d5e54f0716990307a7a2830a18632b40ca566943: Status 404 returned error can't find the container with id 23a322eb3a4ecfdfe6a83e59d5e54f0716990307a7a2830a18632b40ca566943 Oct 30 12:13:30.113446 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.109498 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-dfd96f0e-openshift-default-dockercfg-sq2qj\"" Oct 30 12:13:30.113446 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.113014 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h"] Oct 30 12:13:30.157442 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.157416 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb"] Oct 30 12:13:30.160299 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:13:30.160269 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0ce46a_bfff_4509_97ae_58fca797cc56.slice/crio-560cac31d34f6d6b6226052e6baa35813b7979d0e590e998de8aca9d870f014a WatchSource:0}: Error finding container 560cac31d34f6d6b6226052e6baa35813b7979d0e590e998de8aca9d870f014a: Status 404 returned error can't find the container with id 560cac31d34f6d6b6226052e6baa35813b7979d0e590e998de8aca9d870f014a Oct 30 12:13:30.233682 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.230979 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-token\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.233682 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.231019 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-workload-socket\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.233682 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.231043 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-credential-socket\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.233682 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.231100 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1338306-3768-445a-a6eb-57eafbbeeed0-istiod-ca-cert\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.233682 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.231124 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-envoy\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.233682 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.231139 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-data\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.233682 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.231191 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-workload-certs\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.233682 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.231265 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgwfp\" (UniqueName: \"kubernetes.io/projected/f1338306-3768-445a-a6eb-57eafbbeeed0-kube-api-access-dgwfp\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.233682 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.231374 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-podinfo\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.332875 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.332777 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-token\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.332875 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.332824 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-workload-socket\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.333117 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.333032 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-credential-socket\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.333195 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.333137 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1338306-3768-445a-a6eb-57eafbbeeed0-istiod-ca-cert\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.333195 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.333172 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-envoy\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.333309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.333284 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-workload-socket\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.333365 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.333339 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-data\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.333415 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.333391 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-credential-socket\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.333468 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.333429 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-workload-certs\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.333523 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.333474 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgwfp\" (UniqueName: \"kubernetes.io/projected/f1338306-3768-445a-a6eb-57eafbbeeed0-kube-api-access-dgwfp\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.333523 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.333515 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-podinfo\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.333688 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.333641 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-data\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.333810 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.333775 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-workload-certs\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.333888 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.333845 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1338306-3768-445a-a6eb-57eafbbeeed0-istiod-ca-cert\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.335434 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.335413 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-envoy\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.335853 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.335832 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-podinfo\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.355388 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.355345 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-token\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.355534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.355495 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgwfp\" (UniqueName: \"kubernetes.io/projected/f1338306-3768-445a-a6eb-57eafbbeeed0-kube-api-access-dgwfp\") pod \"e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.414309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.414276 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:30.742209 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.742185 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h"] Oct 30 12:13:30.743387 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:13:30.743357 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1338306_3768_445a_a6eb_57eafbbeeed0.slice/crio-0d59d2976c5e113eeb1db1aa17c2fbb34565c7d0b73c71fefc5aa440cd4b0845 WatchSource:0}: Error finding container 0d59d2976c5e113eeb1db1aa17c2fbb34565c7d0b73c71fefc5aa440cd4b0845: Status 404 returned error can't find the container with id 0d59d2976c5e113eeb1db1aa17c2fbb34565c7d0b73c71fefc5aa440cd4b0845 Oct 30 12:13:30.745870 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.745835 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:13:30.745958 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.745902 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:13:30.745958 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.745951 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:13:30.868522 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.868316 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb" event={"ID":"cc0ce46a-bfff-4509-97ae-58fca797cc56","Type":"ContainerStarted","Data":"92f639ede901610b7c4ade09ac9e7c268ea1842a58f86b1898a839bcaf0c6171"} Oct 30 12:13:30.868522 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.868424 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb" event={"ID":"cc0ce46a-bfff-4509-97ae-58fca797cc56","Type":"ContainerStarted","Data":"560cac31d34f6d6b6226052e6baa35813b7979d0e590e998de8aca9d870f014a"} Oct 30 12:13:30.869526 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.869500 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb" Oct 30 12:13:30.870989 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.870963 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" event={"ID":"f1338306-3768-445a-a6eb-57eafbbeeed0","Type":"ContainerStarted","Data":"a3e2fd880efa5ac2f6c9d8bf7cd05fc9920a720141df55c85cf00a034d3a339e"} Oct 30 12:13:30.871125 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.871014 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" event={"ID":"f1338306-3768-445a-a6eb-57eafbbeeed0","Type":"ContainerStarted","Data":"0d59d2976c5e113eeb1db1aa17c2fbb34565c7d0b73c71fefc5aa440cd4b0845"} Oct 30 12:13:30.872590 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.872560 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7" event={"ID":"20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb","Type":"ContainerStarted","Data":"c3e804419fc8e573125ca5bb784d0f29133c917e385f8ba7ba5c18c69a6c237f"} Oct 30 12:13:30.872722 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.872611 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7" event={"ID":"20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb","Type":"ContainerStarted","Data":"23a322eb3a4ecfdfe6a83e59d5e54f0716990307a7a2830a18632b40ca566943"} Oct 30 12:13:30.872722 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.872642 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7" Oct 30 12:13:30.911060 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.911000 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb" podStartSLOduration=1.910950155 podStartE2EDuration="1.910950155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:30.908188159 +0000 UTC m=+1161.942326555" watchObservedRunningTime="2026-10-30 12:13:30.910950155 +0000 UTC m=+1161.945088552" Oct 30 12:13:30.957085 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:30.957009 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7" podStartSLOduration=1.956990851 podStartE2EDuration="1.956990851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:30.955865479 +0000 UTC m=+1161.990003898" watchObservedRunningTime="2026-10-30 12:13:30.956990851 +0000 UTC m=+1161.991129248" Oct 30 12:13:31.001159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.001042 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" podStartSLOduration=1.001024122 podStartE2EDuration="1.001024122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:31.000021902 +0000 UTC m=+1162.034160322" watchObservedRunningTime="2026-10-30 12:13:31.001024122 +0000 UTC m=+1162.035162592" Oct 30 12:13:31.119444 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.119411 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf"] Oct 30 12:13:31.123509 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.123485 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf" Oct 30 12:13:31.141844 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.141809 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf"] Oct 30 12:13:31.142245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.142215 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e636da41\"" Oct 30 12:13:31.248181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.244130 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7x5f\" (UniqueName: \"kubernetes.io/projected/4e4971c4-9bf1-4140-a050-2d6b84805b2d-kube-api-access-j7x5f\") pod \"payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf\" (UID: \"4e4971c4-9bf1-4140-a050-2d6b84805b2d\") " pod="openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf" Oct 30 12:13:31.248181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.244292 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4971c4-9bf1-4140-a050-2d6b84805b2d-plugins-config-volume\") pod \"payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf\" (UID: \"4e4971c4-9bf1-4140-a050-2d6b84805b2d\") " pod="openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf" Oct 30 12:13:31.345995 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.345872 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4971c4-9bf1-4140-a050-2d6b84805b2d-plugins-config-volume\") pod \"payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf\" (UID: \"4e4971c4-9bf1-4140-a050-2d6b84805b2d\") " pod="openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf" Oct 30 12:13:31.345995 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.345969 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7x5f\" (UniqueName: \"kubernetes.io/projected/4e4971c4-9bf1-4140-a050-2d6b84805b2d-kube-api-access-j7x5f\") pod \"payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf\" (UID: \"4e4971c4-9bf1-4140-a050-2d6b84805b2d\") " pod="openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf" Oct 30 12:13:31.346617 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.346597 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4971c4-9bf1-4140-a050-2d6b84805b2d-plugins-config-volume\") pod \"payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf\" (UID: \"4e4971c4-9bf1-4140-a050-2d6b84805b2d\") " pod="openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf" Oct 30 12:13:31.373821 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.372620 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7x5f\" (UniqueName: \"kubernetes.io/projected/4e4971c4-9bf1-4140-a050-2d6b84805b2d-kube-api-access-j7x5f\") pod \"payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf\" (UID: \"4e4971c4-9bf1-4140-a050-2d6b84805b2d\") " pod="openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf" Oct 30 12:13:31.415359 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.415318 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:31.420901 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.420865 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:31.437924 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.437892 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf" Oct 30 12:13:31.741622 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.741591 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf"] Oct 30 12:13:31.743018 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:13:31.742986 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4971c4_9bf1_4140_a050_2d6b84805b2d.slice/crio-aeefe7d42e58d2d66e8386c22ab59ba8b1ab752b07f4c9defdf54fd3bcfe6295 WatchSource:0}: Error finding container aeefe7d42e58d2d66e8386c22ab59ba8b1ab752b07f4c9defdf54fd3bcfe6295: Status 404 returned error can't find the container with id aeefe7d42e58d2d66e8386c22ab59ba8b1ab752b07f4c9defdf54fd3bcfe6295 Oct 30 12:13:31.882001 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.881964 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf" event={"ID":"4e4971c4-9bf1-4140-a050-2d6b84805b2d","Type":"ContainerStarted","Data":"7bd9b950cb190bfeaaf1dece41bfb6c2d6fd43a80530cd946a4ecc881a6ec02d"} Oct 30 12:13:31.882512 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.882029 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf" event={"ID":"4e4971c4-9bf1-4140-a050-2d6b84805b2d","Type":"ContainerStarted","Data":"aeefe7d42e58d2d66e8386c22ab59ba8b1ab752b07f4c9defdf54fd3bcfe6295"} Oct 30 12:13:31.882512 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.882472 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:31.882615 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.882542 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf" Oct 30 12:13:31.883693 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.883670 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:13:31.936094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:31.927845 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf" podStartSLOduration=0.927822662 podStartE2EDuration="927.822662ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:31.921042259 +0000 UTC m=+1162.955180657" watchObservedRunningTime="2026-10-30 12:13:31.927822662 +0000 UTC m=+1162.961961058" Oct 30 12:13:34.875772 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:34.875720 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27"] Oct 30 12:13:34.879660 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:34.879636 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27" Oct 30 12:13:34.895230 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:34.895199 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-dfd96f0e\"" Oct 30 12:13:34.906702 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:34.906670 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-dfd96f0e-dockercfg-mzkd7\"" Oct 30 12:13:34.909171 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:34.909137 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27"] Oct 30 12:13:34.935969 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:34.935933 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2"] Oct 30 12:13:34.939996 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:34.939968 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2" Oct 30 12:13:34.958012 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:34.957980 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2"] Oct 30 12:13:34.983787 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:34.983752 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv6dw\" (UniqueName: \"kubernetes.io/projected/60230ddf-11ad-4c9d-b49e-9d2547dc929e-kube-api-access-vv6dw\") pod \"payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27\" (UID: \"60230ddf-11ad-4c9d-b49e-9d2547dc929e\") " pod="openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27" Oct 30 12:13:34.983975 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:34.983801 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60230ddf-11ad-4c9d-b49e-9d2547dc929e-plugins-config-volume\") pod \"payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27\" (UID: \"60230ddf-11ad-4c9d-b49e-9d2547dc929e\") " pod="openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27" Oct 30 12:13:35.085092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.084848 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6dw\" (UniqueName: \"kubernetes.io/projected/60230ddf-11ad-4c9d-b49e-9d2547dc929e-kube-api-access-vv6dw\") pod \"payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27\" (UID: \"60230ddf-11ad-4c9d-b49e-9d2547dc929e\") " pod="openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27" Oct 30 12:13:35.085092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.084923 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f409f1-8fa9-438b-b05e-128b59362c24-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2\" (UID: \"28f409f1-8fa9-438b-b05e-128b59362c24\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2" Oct 30 12:13:35.085092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.084972 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60230ddf-11ad-4c9d-b49e-9d2547dc929e-plugins-config-volume\") pod \"payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27\" (UID: \"60230ddf-11ad-4c9d-b49e-9d2547dc929e\") " pod="openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27" Oct 30 12:13:35.085092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.085009 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mxlj\" (UniqueName: \"kubernetes.io/projected/28f409f1-8fa9-438b-b05e-128b59362c24-kube-api-access-2mxlj\") pod \"payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2\" (UID: \"28f409f1-8fa9-438b-b05e-128b59362c24\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2" Oct 30 12:13:35.086455 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.086418 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60230ddf-11ad-4c9d-b49e-9d2547dc929e-plugins-config-volume\") pod \"payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27\" (UID: \"60230ddf-11ad-4c9d-b49e-9d2547dc929e\") " pod="openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27" Oct 30 12:13:35.108304 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.108227 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv6dw\" (UniqueName: \"kubernetes.io/projected/60230ddf-11ad-4c9d-b49e-9d2547dc929e-kube-api-access-vv6dw\") pod \"payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27\" (UID: \"60230ddf-11ad-4c9d-b49e-9d2547dc929e\") " pod="openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27" Oct 30 12:13:35.186051 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.186011 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f409f1-8fa9-438b-b05e-128b59362c24-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2\" (UID: \"28f409f1-8fa9-438b-b05e-128b59362c24\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2" Oct 30 12:13:35.186245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.186064 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mxlj\" (UniqueName: \"kubernetes.io/projected/28f409f1-8fa9-438b-b05e-128b59362c24-kube-api-access-2mxlj\") pod \"payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2\" (UID: \"28f409f1-8fa9-438b-b05e-128b59362c24\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2" Oct 30 12:13:35.186797 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.186766 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f409f1-8fa9-438b-b05e-128b59362c24-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2\" (UID: \"28f409f1-8fa9-438b-b05e-128b59362c24\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2" Oct 30 12:13:35.192774 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.192746 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27" Oct 30 12:13:35.208791 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.208763 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mxlj\" (UniqueName: \"kubernetes.io/projected/28f409f1-8fa9-438b-b05e-128b59362c24-kube-api-access-2mxlj\") pod \"payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2\" (UID: \"28f409f1-8fa9-438b-b05e-128b59362c24\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2" Oct 30 12:13:35.254655 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.254565 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2" Oct 30 12:13:35.493676 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.493646 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27"] Oct 30 12:13:35.494261 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:13:35.494234 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60230ddf_11ad_4c9d_b49e_9d2547dc929e.slice/crio-c6ee14ea062f8240dcf3819e4ac1d3a6b928b9cbdd8a2ac86e14810a07cbd3f9 WatchSource:0}: Error finding container c6ee14ea062f8240dcf3819e4ac1d3a6b928b9cbdd8a2ac86e14810a07cbd3f9: Status 404 returned error can't find the container with id c6ee14ea062f8240dcf3819e4ac1d3a6b928b9cbdd8a2ac86e14810a07cbd3f9 Oct 30 12:13:35.583766 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.583741 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2"] Oct 30 12:13:35.584612 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:13:35.584588 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f409f1_8fa9_438b_b05e_128b59362c24.slice/crio-5c0de58ca642ab415520317f719353f26d402be1a782f008f85fca378e5e6740 WatchSource:0}: Error finding container 5c0de58ca642ab415520317f719353f26d402be1a782f008f85fca378e5e6740: Status 404 returned error can't find the container with id 5c0de58ca642ab415520317f719353f26d402be1a782f008f85fca378e5e6740 Oct 30 12:13:35.904908 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.904797 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27" event={"ID":"60230ddf-11ad-4c9d-b49e-9d2547dc929e","Type":"ContainerStarted","Data":"e6534cde6cddf7149fca7024cb021cf9bd9e77567ec13507ee9811583ba18bac"} Oct 30 12:13:35.904908 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.904852 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27" event={"ID":"60230ddf-11ad-4c9d-b49e-9d2547dc929e","Type":"ContainerStarted","Data":"c6ee14ea062f8240dcf3819e4ac1d3a6b928b9cbdd8a2ac86e14810a07cbd3f9"} Oct 30 12:13:35.904908 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.904880 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27" Oct 30 12:13:35.906437 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.906411 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2" event={"ID":"28f409f1-8fa9-438b-b05e-128b59362c24","Type":"ContainerStarted","Data":"3c69b7daa2868ab216112bb2f0b89d4c0f17db8f62f1995310b30e8723311837"} Oct 30 12:13:35.906564 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.906452 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2" event={"ID":"28f409f1-8fa9-438b-b05e-128b59362c24","Type":"ContainerStarted","Data":"5c0de58ca642ab415520317f719353f26d402be1a782f008f85fca378e5e6740"} Oct 30 12:13:35.906564 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.906491 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2" Oct 30 12:13:35.944631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.944575 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27" podStartSLOduration=1.944560313 podStartE2EDuration="1.944560313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:35.943714291 +0000 UTC m=+1166.977852687" watchObservedRunningTime="2026-10-30 12:13:35.944560313 +0000 UTC m=+1166.978698709" Oct 30 12:13:35.983715 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:35.983663 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2" podStartSLOduration=1.983648855 podStartE2EDuration="1.983648855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:35.983523989 +0000 UTC m=+1167.017662385" watchObservedRunningTime="2026-10-30 12:13:35.983648855 +0000 UTC m=+1167.017787250" Oct 30 12:13:38.842558 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:38.842524 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6" Oct 30 12:13:38.842970 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:38.842599 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6" Oct 30 12:13:39.854626 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:39.854594 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p" Oct 30 12:13:39.855101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:39.854676 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t" Oct 30 12:13:41.148495 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.148451 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4"] Oct 30 12:13:41.154361 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.154330 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" Oct 30 12:13:41.202401 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.192880 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rtvpm\"" Oct 30 12:13:41.202401 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.198569 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 12:13:41.202401 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.199125 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-dfd96f0e-serving-cert\"" Oct 30 12:13:41.297276 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.295520 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4"] Oct 30 12:13:41.297276 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.296251 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt9cw\" (UniqueName: \"kubernetes.io/projected/2b142407-8432-4ed2-bb39-2744274f83b5-kube-api-access-lt9cw\") pod \"maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4\" (UID: \"2b142407-8432-4ed2-bb39-2744274f83b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" Oct 30 12:13:41.297276 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.296295 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b142407-8432-4ed2-bb39-2744274f83b5-maas-api-tls\") pod \"maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4\" (UID: \"2b142407-8432-4ed2-bb39-2744274f83b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" Oct 30 12:13:41.297276 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.296323 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b142407-8432-4ed2-bb39-2744274f83b5-openshift-service-ca\") pod \"maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4\" (UID: \"2b142407-8432-4ed2-bb39-2744274f83b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" Oct 30 12:13:41.297276 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.296356 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b142407-8432-4ed2-bb39-2744274f83b5-metrics-certs\") pod \"maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4\" (UID: \"2b142407-8432-4ed2-bb39-2744274f83b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" Oct 30 12:13:41.397857 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.397802 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9cw\" (UniqueName: \"kubernetes.io/projected/2b142407-8432-4ed2-bb39-2744274f83b5-kube-api-access-lt9cw\") pod \"maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4\" (UID: \"2b142407-8432-4ed2-bb39-2744274f83b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" Oct 30 12:13:41.398061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.397866 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b142407-8432-4ed2-bb39-2744274f83b5-maas-api-tls\") pod \"maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4\" (UID: \"2b142407-8432-4ed2-bb39-2744274f83b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" Oct 30 12:13:41.398061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.397896 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b142407-8432-4ed2-bb39-2744274f83b5-openshift-service-ca\") pod \"maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4\" (UID: \"2b142407-8432-4ed2-bb39-2744274f83b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" Oct 30 12:13:41.398061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.397927 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b142407-8432-4ed2-bb39-2744274f83b5-metrics-certs\") pod \"maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4\" (UID: \"2b142407-8432-4ed2-bb39-2744274f83b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" Oct 30 12:13:41.398061 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:13:41.397986 4468 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-serving-cert: secret "maas-api-e2e-derive-dfd96f0e-serving-cert" not found Oct 30 12:13:41.398061 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:13:41.398060 4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b142407-8432-4ed2-bb39-2744274f83b5-maas-api-tls podName:2b142407-8432-4ed2-bb39-2744274f83b5 nodeName:}" failed. No retries permitted until 2026-10-30 12:13:41.898039672 +0000 UTC m=+1172.932178052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2b142407-8432-4ed2-bb39-2744274f83b5-maas-api-tls") pod "maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" (UID: "2b142407-8432-4ed2-bb39-2744274f83b5") : secret "maas-api-e2e-derive-dfd96f0e-serving-cert" not found Oct 30 12:13:41.398695 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.398636 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b142407-8432-4ed2-bb39-2744274f83b5-openshift-service-ca\") pod \"maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4\" (UID: \"2b142407-8432-4ed2-bb39-2744274f83b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" Oct 30 12:13:41.401317 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.401287 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b142407-8432-4ed2-bb39-2744274f83b5-metrics-certs\") pod \"maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4\" (UID: \"2b142407-8432-4ed2-bb39-2744274f83b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" Oct 30 12:13:41.437791 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.437748 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt9cw\" (UniqueName: \"kubernetes.io/projected/2b142407-8432-4ed2-bb39-2744274f83b5-kube-api-access-lt9cw\") pod \"maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4\" (UID: \"2b142407-8432-4ed2-bb39-2744274f83b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" Oct 30 12:13:41.886290 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.886259 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7" Oct 30 12:13:41.924094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.923798 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b142407-8432-4ed2-bb39-2744274f83b5-maas-api-tls\") pod \"maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4\" (UID: \"2b142407-8432-4ed2-bb39-2744274f83b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" Oct 30 12:13:41.932095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.931990 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b142407-8432-4ed2-bb39-2744274f83b5-maas-api-tls\") pod \"maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4\" (UID: \"2b142407-8432-4ed2-bb39-2744274f83b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" Oct 30 12:13:41.934703 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:41.934651 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" Oct 30 12:13:42.261897 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:42.261864 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4"] Oct 30 12:13:42.262481 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:13:42.262443 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b142407_8432_4ed2_bb39_2744274f83b5.slice/crio-e45efaa945c5ba9b2ad4fbab0075cfac2f9af7c5153d551195ddcd5115efec6d WatchSource:0}: Error finding container e45efaa945c5ba9b2ad4fbab0075cfac2f9af7c5153d551195ddcd5115efec6d: Status 404 returned error can't find the container with id e45efaa945c5ba9b2ad4fbab0075cfac2f9af7c5153d551195ddcd5115efec6d Oct 30 12:13:42.891681 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:42.891650 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb" Oct 30 12:13:42.976942 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:42.976839 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" event={"ID":"2b142407-8432-4ed2-bb39-2744274f83b5","Type":"ContainerStarted","Data":"f76162d27fb7d081ef59b5c96a6fc57848dd7eae2f3431ba7bb8f7bdb8a92e55"} Oct 30 12:13:42.976942 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:42.976900 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" event={"ID":"2b142407-8432-4ed2-bb39-2744274f83b5","Type":"ContainerStarted","Data":"e45efaa945c5ba9b2ad4fbab0075cfac2f9af7c5153d551195ddcd5115efec6d"} Oct 30 12:13:42.977714 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:42.977693 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" Oct 30 12:13:43.035591 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:43.035524 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" podStartSLOduration=1.609683992 podStartE2EDuration="2.035502919s" totalImagesPullingTime="425.818927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:43.033509948 +0000 UTC m=+1174.067648345" watchObservedRunningTime="2026-10-30 12:13:43.035502919 +0000 UTC m=+1174.069641316" Oct 30 12:13:43.896353 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:43.896322 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf" Oct 30 12:13:46.938326 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:46.938287 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2" Oct 30 12:13:46.945085 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:46.944187 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27" Oct 30 12:13:48.061339 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:48.061246 4468 generic.go:356] "Generic (PLEG): container finished" podID="2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" containerID="544320268b1126839dc8d7b3f0e7037a30d6171d202513f4bd6842b2a280f3d3" exitCode=0 Oct 30 12:13:48.061888 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:48.061325 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" event={"ID":"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b","Type":"ContainerDied","Data":"544320268b1126839dc8d7b3f0e7037a30d6171d202513f4bd6842b2a280f3d3"} Oct 30 12:13:48.061888 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:48.061860 4468 scope.go:122] "RemoveContainer" containerID="544320268b1126839dc8d7b3f0e7037a30d6171d202513f4bd6842b2a280f3d3" Oct 30 12:13:48.063242 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:48.063215 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:13:48.063350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:48.063296 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:13:48.063414 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:48.063344 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:13:49.070207 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:49.070167 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" event={"ID":"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b","Type":"ContainerStarted","Data":"569fb20fcf136fac6307d7aafee9099a009a851dc62a4f8385b9c2d22f58c4fd"} Oct 30 12:13:49.546703 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:49.546660 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:49.546887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:49.546842 4468 patch_prober.go:29] interesting pod/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 12:13:49.546963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:49.546889 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" podUID="2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 12:13:49.994004 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:49.993963 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" Oct 30 12:13:50.547025 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:50.546982 4468 patch_prober.go:29] interesting pod/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 12:13:50.547530 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:50.547056 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" podUID="2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 12:13:51.548305 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:51.547958 4468 patch_prober.go:29] interesting pod/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 12:13:51.548305 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:51.548007 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" podUID="2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 12:13:51.739092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:51.738318 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6"] Oct 30 12:13:51.771422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:51.771386 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6"] Oct 30 12:13:52.546976 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:52.546930 4468 patch_prober.go:29] interesting pod/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 12:13:52.548214 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:52.546999 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" podUID="2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 12:13:53.132241 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:53.132206 4468 generic.go:356] "Generic (PLEG): container finished" podID="d5e74d16-3dde-444b-8f10-769314d02026" containerID="0fc49bd91096e7afd1d01a24d6e748edc6b1419d2c8358803cb1ff7ca5458a01" exitCode=0 Oct 30 12:13:53.132682 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:53.132295 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" event={"ID":"d5e74d16-3dde-444b-8f10-769314d02026","Type":"ContainerDied","Data":"0fc49bd91096e7afd1d01a24d6e748edc6b1419d2c8358803cb1ff7ca5458a01"} Oct 30 12:13:53.132682 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:53.132652 4468 scope.go:122] "RemoveContainer" containerID="0fc49bd91096e7afd1d01a24d6e748edc6b1419d2c8358803cb1ff7ca5458a01" Oct 30 12:13:53.134097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:53.134046 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:13:53.134223 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:53.134168 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:13:53.134287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:53.134222 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:13:53.547239 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:53.547208 4468 patch_prober.go:29] interesting pod/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 12:13:53.547401 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:53.547266 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" podUID="2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 12:13:53.615093 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:13:53.611789 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:13:53.824923 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:53.824825 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:54.140737 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:54.140635 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" event={"ID":"d5e74d16-3dde-444b-8f10-769314d02026","Type":"ContainerStarted","Data":"4e3b062f132b0f27398ad9fb98648ed43cebb0411e78434444910df4fdf4427f"} Oct 30 12:13:54.331095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:54.328751 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:13:54.331095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:54.328907 4468 patch_prober.go:29] interesting pod/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 12:13:54.331095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:54.328954 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" podUID="d5e74d16-3dde-444b-8f10-769314d02026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 12:13:54.550935 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:54.550904 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:54.551306 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:54.551286 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:54.552131 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:54.552116 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:13:55.314629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:55.314534 4468 patch_prober.go:29] interesting pod/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 12:13:55.315286 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:55.315253 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" podUID="d5e74d16-3dde-444b-8f10-769314d02026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 12:13:56.327097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:56.322621 4468 patch_prober.go:29] interesting pod/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 12:13:56.327097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:56.322718 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" podUID="d5e74d16-3dde-444b-8f10-769314d02026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 12:13:56.739245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:56.739154 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6" podUID="f581c0ea-a55a-4a4a-b892-45c8759ba074" containerName="payload-processing" containerID="cri-o://a8c0ae2173bff9b55a7a668059f10ad97b888e8f1895a3ea63f06bba636e76ce" gracePeriod=25 Oct 30 12:13:56.772321 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:56.772275 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6" podUID="dac85f8a-9ce0-449f-b7a3-277c53941fd0" containerName="payload-pre-processing" containerID="cri-o://782a0b7cba571be37b744350916cae871058901c399c0cecdc542bc91475393c" gracePeriod=25 Oct 30 12:13:57.177602 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.177572 4468 generic.go:356] "Generic (PLEG): container finished" podID="f581c0ea-a55a-4a4a-b892-45c8759ba074" containerID="a8c0ae2173bff9b55a7a668059f10ad97b888e8f1895a3ea63f06bba636e76ce" exitCode=0 Oct 30 12:13:57.177971 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.177732 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6" event={"ID":"f581c0ea-a55a-4a4a-b892-45c8759ba074","Type":"ContainerDied","Data":"a8c0ae2173bff9b55a7a668059f10ad97b888e8f1895a3ea63f06bba636e76ce"} Oct 30 12:13:57.179430 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.179338 4468 generic.go:356] "Generic (PLEG): container finished" podID="dac85f8a-9ce0-449f-b7a3-277c53941fd0" containerID="782a0b7cba571be37b744350916cae871058901c399c0cecdc542bc91475393c" exitCode=0 Oct 30 12:13:57.179430 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.179386 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6" event={"ID":"dac85f8a-9ce0-449f-b7a3-277c53941fd0","Type":"ContainerDied","Data":"782a0b7cba571be37b744350916cae871058901c399c0cecdc542bc91475393c"} Oct 30 12:13:57.199974 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.199945 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6" Oct 30 12:13:57.287311 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.287285 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6" Oct 30 12:13:57.293141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.293112 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f581c0ea-a55a-4a4a-b892-45c8759ba074-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f581c0ea-a55a-4a4a-b892-45c8759ba074-plugins-config-volume\") pod \"f581c0ea-a55a-4a4a-b892-45c8759ba074\" (UID: \"f581c0ea-a55a-4a4a-b892-45c8759ba074\") " Oct 30 12:13:57.293141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.293146 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f581c0ea-a55a-4a4a-b892-45c8759ba074-kube-api-access-x86x6\" (UniqueName: \"kubernetes.io/projected/f581c0ea-a55a-4a4a-b892-45c8759ba074-kube-api-access-x86x6\") pod \"f581c0ea-a55a-4a4a-b892-45c8759ba074\" (UID: \"f581c0ea-a55a-4a4a-b892-45c8759ba074\") " Oct 30 12:13:57.293653 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.293622 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f581c0ea-a55a-4a4a-b892-45c8759ba074-plugins-config-volume" pod "f581c0ea-a55a-4a4a-b892-45c8759ba074" (UID: "f581c0ea-a55a-4a4a-b892-45c8759ba074"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:13:57.295391 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.295369 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f581c0ea-a55a-4a4a-b892-45c8759ba074-kube-api-access-x86x6" pod "f581c0ea-a55a-4a4a-b892-45c8759ba074" (UID: "f581c0ea-a55a-4a4a-b892-45c8759ba074"). InnerVolumeSpecName "kube-api-access-x86x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:13:57.332706 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.332661 4468 patch_prober.go:29] interesting pod/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 12:13:57.353002 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.352897 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" podUID="d5e74d16-3dde-444b-8f10-769314d02026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 12:13:57.394473 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.394054 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac85f8a-9ce0-449f-b7a3-277c53941fd0-kube-api-access-wv2j7\" (UniqueName: \"kubernetes.io/projected/dac85f8a-9ce0-449f-b7a3-277c53941fd0-kube-api-access-wv2j7\") pod \"dac85f8a-9ce0-449f-b7a3-277c53941fd0\" (UID: \"dac85f8a-9ce0-449f-b7a3-277c53941fd0\") " Oct 30 12:13:57.394473 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.394263 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dac85f8a-9ce0-449f-b7a3-277c53941fd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac85f8a-9ce0-449f-b7a3-277c53941fd0-plugins-config-volume\") pod \"dac85f8a-9ce0-449f-b7a3-277c53941fd0\" (UID: \"dac85f8a-9ce0-449f-b7a3-277c53941fd0\") " Oct 30 12:13:57.394473 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.394466 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f581c0ea-a55a-4a4a-b892-45c8759ba074-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:13:57.394473 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.394478 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x86x6\" (UniqueName: \"kubernetes.io/projected/f581c0ea-a55a-4a4a-b892-45c8759ba074-kube-api-access-x86x6\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:13:57.394777 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.394727 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac85f8a-9ce0-449f-b7a3-277c53941fd0-plugins-config-volume" pod "dac85f8a-9ce0-449f-b7a3-277c53941fd0" (UID: "dac85f8a-9ce0-449f-b7a3-277c53941fd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:13:57.396668 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.396643 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac85f8a-9ce0-449f-b7a3-277c53941fd0-kube-api-access-wv2j7" pod "dac85f8a-9ce0-449f-b7a3-277c53941fd0" (UID: "dac85f8a-9ce0-449f-b7a3-277c53941fd0"). InnerVolumeSpecName "kube-api-access-wv2j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:13:57.457330 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.457280 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7"] Oct 30 12:13:57.490160 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.490118 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb"] Oct 30 12:13:57.495313 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.495281 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac85f8a-9ce0-449f-b7a3-277c53941fd0-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:13:57.495313 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:57.495309 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv2j7\" (UniqueName: \"kubernetes.io/projected/dac85f8a-9ce0-449f-b7a3-277c53941fd0-kube-api-access-wv2j7\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:13:58.196688 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:58.196646 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6" event={"ID":"dac85f8a-9ce0-449f-b7a3-277c53941fd0","Type":"ContainerDied","Data":"a78358f9cd5c48c16288df39c8b906ac0913341c4d9d7a263920ad33867f060d"} Oct 30 12:13:58.199283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:58.199255 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6" Oct 30 12:13:58.204094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:58.199302 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6" event={"ID":"f581c0ea-a55a-4a4a-b892-45c8759ba074","Type":"ContainerDied","Data":"fcbe24466bf84b2ba965a1c6e24b958fc2928d1aa67f4eafe6952bdace6097e2"} Oct 30 12:13:58.204094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:58.199326 4468 scope.go:122] "RemoveContainer" containerID="782a0b7cba571be37b744350916cae871058901c399c0cecdc542bc91475393c" Oct 30 12:13:58.204094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:58.196998 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6" Oct 30 12:13:58.316971 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:58.316931 4468 patch_prober.go:29] interesting pod/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 12:13:58.317266 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:58.317238 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" podUID="d5e74d16-3dde-444b-8f10-769314d02026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 12:13:58.370349 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:58.370319 4468 scope.go:122] "RemoveContainer" containerID="a8c0ae2173bff9b55a7a668059f10ad97b888e8f1895a3ea63f06bba636e76ce" Oct 30 12:13:58.374820 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:58.374762 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6"] Oct 30 12:13:58.377113 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:58.377084 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-307283d2-76cb675ff5-hqzw6"] Oct 30 12:13:58.445112 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:58.438878 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6"] Oct 30 12:13:58.474107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:58.473991 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-307283d2-88bc6f9d6-jtqk6"] Oct 30 12:13:59.089228 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:59.089184 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dac85f8a-9ce0-449f-b7a3-277c53941fd0" path="/var/lib/kubelet/pods/dac85f8a-9ce0-449f-b7a3-277c53941fd0/volumes" Oct 30 12:13:59.089656 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:59.089637 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f581c0ea-a55a-4a4a-b892-45c8759ba074" path="/var/lib/kubelet/pods/f581c0ea-a55a-4a4a-b892-45c8759ba074/volumes" Oct 30 12:13:59.326095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:59.315510 4468 patch_prober.go:29] interesting pod/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 12:13:59.326095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:13:59.315583 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" podUID="d5e74d16-3dde-444b-8f10-769314d02026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 12:14:00.365810 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:00.365256 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:14:00.366279 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:00.366261 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:14:00.367476 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:00.367455 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:14:01.117334 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:01.117247 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t"] Oct 30 12:14:01.148021 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:01.147976 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p"] Oct 30 12:14:02.458888 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:02.458842 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7" podUID="20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb" containerName="payload-processing" containerID="cri-o://c3e804419fc8e573125ca5bb784d0f29133c917e385f8ba7ba5c18c69a6c237f" gracePeriod=25 Oct 30 12:14:02.490482 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:02.490408 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb" podUID="cc0ce46a-bfff-4509-97ae-58fca797cc56" containerName="payload-pre-processing" containerID="cri-o://92f639ede901610b7c4ade09ac9e7c268ea1842a58f86b1898a839bcaf0c6171" gracePeriod=25 Oct 30 12:14:02.795113 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:02.793653 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7" Oct 30 12:14:02.830843 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:02.830817 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb" Oct 30 12:14:02.891199 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:02.891148 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb-plugins-config-volume\") pod \"20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb\" (UID: \"20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb\") " Oct 30 12:14:02.891389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:02.891224 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb-kube-api-access-njf8v\" (UniqueName: \"kubernetes.io/projected/20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb-kube-api-access-njf8v\") pod \"20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb\" (UID: \"20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb\") " Oct 30 12:14:02.891702 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:02.891669 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb-plugins-config-volume" pod "20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb" (UID: "20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:14:02.893574 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:02.893544 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb-kube-api-access-njf8v" pod "20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb" (UID: "20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb"). InnerVolumeSpecName "kube-api-access-njf8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:02.932281 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:02.932236 4468 patch_prober.go:29] interesting pod/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:14:02.932442 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:02.932306 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" podUID="f1338306-3768-445a-a6eb-57eafbbeeed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:14:02.993122 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:02.993018 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc0ce46a-bfff-4509-97ae-58fca797cc56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0ce46a-bfff-4509-97ae-58fca797cc56-plugins-config-volume\") pod \"cc0ce46a-bfff-4509-97ae-58fca797cc56\" (UID: \"cc0ce46a-bfff-4509-97ae-58fca797cc56\") " Oct 30 12:14:02.993287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:02.993193 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0ce46a-bfff-4509-97ae-58fca797cc56-kube-api-access-7xjw4\" (UniqueName: \"kubernetes.io/projected/cc0ce46a-bfff-4509-97ae-58fca797cc56-kube-api-access-7xjw4\") pod \"cc0ce46a-bfff-4509-97ae-58fca797cc56\" (UID: \"cc0ce46a-bfff-4509-97ae-58fca797cc56\") " Oct 30 12:14:02.993435 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:02.993407 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0ce46a-bfff-4509-97ae-58fca797cc56-plugins-config-volume" pod "cc0ce46a-bfff-4509-97ae-58fca797cc56" (UID: "cc0ce46a-bfff-4509-97ae-58fca797cc56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:14:02.993581 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:02.993558 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:02.993643 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:02.993587 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njf8v\" (UniqueName: \"kubernetes.io/projected/20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb-kube-api-access-njf8v\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:02.993643 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:02.993604 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0ce46a-bfff-4509-97ae-58fca797cc56-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:02.995290 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:02.995269 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0ce46a-bfff-4509-97ae-58fca797cc56-kube-api-access-7xjw4" pod "cc0ce46a-bfff-4509-97ae-58fca797cc56" (UID: "cc0ce46a-bfff-4509-97ae-58fca797cc56"). InnerVolumeSpecName "kube-api-access-7xjw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:03.094574 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.094528 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xjw4\" (UniqueName: \"kubernetes.io/projected/cc0ce46a-bfff-4509-97ae-58fca797cc56-kube-api-access-7xjw4\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:03.278756 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.278654 4468 generic.go:356] "Generic (PLEG): container finished" podID="20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb" containerID="c3e804419fc8e573125ca5bb784d0f29133c917e385f8ba7ba5c18c69a6c237f" exitCode=0 Oct 30 12:14:03.278756 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.278699 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7" Oct 30 12:14:03.278984 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.278746 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7" event={"ID":"20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb","Type":"ContainerDied","Data":"c3e804419fc8e573125ca5bb784d0f29133c917e385f8ba7ba5c18c69a6c237f"} Oct 30 12:14:03.278984 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.278813 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7" event={"ID":"20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb","Type":"ContainerDied","Data":"23a322eb3a4ecfdfe6a83e59d5e54f0716990307a7a2830a18632b40ca566943"} Oct 30 12:14:03.278984 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.278848 4468 scope.go:122] "RemoveContainer" containerID="c3e804419fc8e573125ca5bb784d0f29133c917e385f8ba7ba5c18c69a6c237f" Oct 30 12:14:03.280540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.280471 4468 generic.go:356] "Generic (PLEG): container finished" podID="cc0ce46a-bfff-4509-97ae-58fca797cc56" containerID="92f639ede901610b7c4ade09ac9e7c268ea1842a58f86b1898a839bcaf0c6171" exitCode=0 Oct 30 12:14:03.280540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.280504 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb" event={"ID":"cc0ce46a-bfff-4509-97ae-58fca797cc56","Type":"ContainerDied","Data":"92f639ede901610b7c4ade09ac9e7c268ea1842a58f86b1898a839bcaf0c6171"} Oct 30 12:14:03.280711 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.280573 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb" event={"ID":"cc0ce46a-bfff-4509-97ae-58fca797cc56","Type":"ContainerDied","Data":"560cac31d34f6d6b6226052e6baa35813b7979d0e590e998de8aca9d870f014a"} Oct 30 12:14:03.280711 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.280517 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb" Oct 30 12:14:03.290033 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.290012 4468 scope.go:122] "RemoveContainer" containerID="c3e804419fc8e573125ca5bb784d0f29133c917e385f8ba7ba5c18c69a6c237f" Oct 30 12:14:03.290446 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:14:03.290420 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e804419fc8e573125ca5bb784d0f29133c917e385f8ba7ba5c18c69a6c237f\": container with ID starting with c3e804419fc8e573125ca5bb784d0f29133c917e385f8ba7ba5c18c69a6c237f not found: ID does not exist" containerID="c3e804419fc8e573125ca5bb784d0f29133c917e385f8ba7ba5c18c69a6c237f" Oct 30 12:14:03.290515 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.290458 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e804419fc8e573125ca5bb784d0f29133c917e385f8ba7ba5c18c69a6c237f"} err="failed to get container status \"c3e804419fc8e573125ca5bb784d0f29133c917e385f8ba7ba5c18c69a6c237f\": rpc error: code = NotFound desc = could not find container \"c3e804419fc8e573125ca5bb784d0f29133c917e385f8ba7ba5c18c69a6c237f\": container with ID starting with c3e804419fc8e573125ca5bb784d0f29133c917e385f8ba7ba5c18c69a6c237f not found: ID does not exist" Oct 30 12:14:03.290515 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.290475 4468 scope.go:122] "RemoveContainer" containerID="92f639ede901610b7c4ade09ac9e7c268ea1842a58f86b1898a839bcaf0c6171" Oct 30 12:14:03.300662 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.300639 4468 scope.go:122] "RemoveContainer" containerID="92f639ede901610b7c4ade09ac9e7c268ea1842a58f86b1898a839bcaf0c6171" Oct 30 12:14:03.301005 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:14:03.300987 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f639ede901610b7c4ade09ac9e7c268ea1842a58f86b1898a839bcaf0c6171\": container with ID starting with 92f639ede901610b7c4ade09ac9e7c268ea1842a58f86b1898a839bcaf0c6171 not found: ID does not exist" containerID="92f639ede901610b7c4ade09ac9e7c268ea1842a58f86b1898a839bcaf0c6171" Oct 30 12:14:03.301119 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.301015 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f639ede901610b7c4ade09ac9e7c268ea1842a58f86b1898a839bcaf0c6171"} err="failed to get container status \"92f639ede901610b7c4ade09ac9e7c268ea1842a58f86b1898a839bcaf0c6171\": rpc error: code = NotFound desc = could not find container \"92f639ede901610b7c4ade09ac9e7c268ea1842a58f86b1898a839bcaf0c6171\": container with ID starting with 92f639ede901610b7c4ade09ac9e7c268ea1842a58f86b1898a839bcaf0c6171 not found: ID does not exist" Oct 30 12:14:03.327792 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.327753 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7"] Oct 30 12:14:03.339509 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.339474 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c85bd66-76dcc7d9d4-scct7"] Oct 30 12:14:03.366396 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.366334 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb"] Oct 30 12:14:03.369261 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.369233 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c85bd66-55bcd769cf-q25kb"] Oct 30 12:14:03.660128 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:03.660061 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf"] Oct 30 12:14:04.308216 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:04.306362 4468 generic.go:356] "Generic (PLEG): container finished" podID="f1338306-3768-445a-a6eb-57eafbbeeed0" containerID="a3e2fd880efa5ac2f6c9d8bf7cd05fc9920a720141df55c85cf00a034d3a339e" exitCode=0 Oct 30 12:14:04.308216 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:04.306575 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" event={"ID":"f1338306-3768-445a-a6eb-57eafbbeeed0","Type":"ContainerDied","Data":"a3e2fd880efa5ac2f6c9d8bf7cd05fc9920a720141df55c85cf00a034d3a339e"} Oct 30 12:14:04.308216 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:04.307295 4468 scope.go:122] "RemoveContainer" containerID="a3e2fd880efa5ac2f6c9d8bf7cd05fc9920a720141df55c85cf00a034d3a339e" Oct 30 12:14:04.308943 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:04.308888 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:14:04.309016 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:04.309000 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:14:04.309112 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:04.309050 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:14:04.391474 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:04.391422 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv"] Oct 30 12:14:04.391705 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:04.391676 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" podUID="2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" containerName="istio-proxy" containerID="cri-o://569fb20fcf136fac6307d7aafee9099a009a851dc62a4f8385b9c2d22f58c4fd" gracePeriod=30 Oct 30 12:14:05.089911 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:05.089878 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb" path="/var/lib/kubelet/pods/20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb/volumes" Oct 30 12:14:05.090376 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:05.090360 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc0ce46a-bfff-4509-97ae-58fca797cc56" path="/var/lib/kubelet/pods/cc0ce46a-bfff-4509-97ae-58fca797cc56/volumes" Oct 30 12:14:05.331164 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:05.331123 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" event={"ID":"f1338306-3768-445a-a6eb-57eafbbeeed0","Type":"ContainerStarted","Data":"3be1247efe812dadc1bcdc144f3b5e4a0f7f67bfeb6faf42f5ba7c92cdff28c2"} Oct 30 12:14:05.423005 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:05.422561 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:14:05.423005 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:05.422693 4468 patch_prober.go:29] interesting pod/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 12:14:05.423005 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:05.422740 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" podUID="f1338306-3768-445a-a6eb-57eafbbeeed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 12:14:06.127094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.124152 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t" podUID="1be0127f-6554-4c65-be90-db22d98a6071" containerName="payload-processing" containerID="cri-o://039b2dcfd1e42bc947771ea02827bc507c472288f973ff2877de7ba2de5cf018" gracePeriod=25 Oct 30 12:14:06.149013 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.148977 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p" podUID="184b5d2e-de0b-4f06-bbb5-37522055a6e8" containerName="payload-pre-processing" containerID="cri-o://b127c61e210cc540d32191a6d5383eb78aea753f7ca44abd3335c5d0f3488a2b" gracePeriod=25 Oct 30 12:14:06.356109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.342152 4468 generic.go:356] "Generic (PLEG): container finished" podID="1be0127f-6554-4c65-be90-db22d98a6071" containerID="039b2dcfd1e42bc947771ea02827bc507c472288f973ff2877de7ba2de5cf018" exitCode=0 Oct 30 12:14:06.356109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.342257 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t" event={"ID":"1be0127f-6554-4c65-be90-db22d98a6071","Type":"ContainerDied","Data":"039b2dcfd1e42bc947771ea02827bc507c472288f973ff2877de7ba2de5cf018"} Oct 30 12:14:06.356109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.345575 4468 generic.go:356] "Generic (PLEG): container finished" podID="184b5d2e-de0b-4f06-bbb5-37522055a6e8" containerID="b127c61e210cc540d32191a6d5383eb78aea753f7ca44abd3335c5d0f3488a2b" exitCode=0 Oct 30 12:14:06.356109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.345909 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p" event={"ID":"184b5d2e-de0b-4f06-bbb5-37522055a6e8","Type":"ContainerDied","Data":"b127c61e210cc540d32191a6d5383eb78aea753f7ca44abd3335c5d0f3488a2b"} Oct 30 12:14:06.429253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.417582 4468 patch_prober.go:29] interesting pod/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 12:14:06.429253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.417643 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" podUID="f1338306-3768-445a-a6eb-57eafbbeeed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 12:14:06.464416 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.464136 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t" Oct 30 12:14:06.543524 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.543490 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p" Oct 30 12:14:06.558102 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.558051 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1be0127f-6554-4c65-be90-db22d98a6071-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be0127f-6554-4c65-be90-db22d98a6071-plugins-config-volume\") pod \"1be0127f-6554-4c65-be90-db22d98a6071\" (UID: \"1be0127f-6554-4c65-be90-db22d98a6071\") " Oct 30 12:14:06.558297 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.558188 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be0127f-6554-4c65-be90-db22d98a6071-kube-api-access-bkjt6\" (UniqueName: \"kubernetes.io/projected/1be0127f-6554-4c65-be90-db22d98a6071-kube-api-access-bkjt6\") pod \"1be0127f-6554-4c65-be90-db22d98a6071\" (UID: \"1be0127f-6554-4c65-be90-db22d98a6071\") " Oct 30 12:14:06.558418 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.558395 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1be0127f-6554-4c65-be90-db22d98a6071-plugins-config-volume" pod "1be0127f-6554-4c65-be90-db22d98a6071" (UID: "1be0127f-6554-4c65-be90-db22d98a6071"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:14:06.558494 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.558478 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be0127f-6554-4c65-be90-db22d98a6071-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:06.560336 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.560310 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be0127f-6554-4c65-be90-db22d98a6071-kube-api-access-bkjt6" pod "1be0127f-6554-4c65-be90-db22d98a6071" (UID: "1be0127f-6554-4c65-be90-db22d98a6071"). InnerVolumeSpecName "kube-api-access-bkjt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:06.669548 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.663633 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184b5d2e-de0b-4f06-bbb5-37522055a6e8-kube-api-access-4d4gl\" (UniqueName: \"kubernetes.io/projected/184b5d2e-de0b-4f06-bbb5-37522055a6e8-kube-api-access-4d4gl\") pod \"184b5d2e-de0b-4f06-bbb5-37522055a6e8\" (UID: \"184b5d2e-de0b-4f06-bbb5-37522055a6e8\") " Oct 30 12:14:06.669548 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.663759 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184b5d2e-de0b-4f06-bbb5-37522055a6e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184b5d2e-de0b-4f06-bbb5-37522055a6e8-plugins-config-volume\") pod \"184b5d2e-de0b-4f06-bbb5-37522055a6e8\" (UID: \"184b5d2e-de0b-4f06-bbb5-37522055a6e8\") " Oct 30 12:14:06.669548 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.664146 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkjt6\" (UniqueName: \"kubernetes.io/projected/1be0127f-6554-4c65-be90-db22d98a6071-kube-api-access-bkjt6\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:06.669865 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.669560 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184b5d2e-de0b-4f06-bbb5-37522055a6e8-plugins-config-volume" pod "184b5d2e-de0b-4f06-bbb5-37522055a6e8" (UID: "184b5d2e-de0b-4f06-bbb5-37522055a6e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:14:06.679064 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.679021 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184b5d2e-de0b-4f06-bbb5-37522055a6e8-kube-api-access-4d4gl" pod "184b5d2e-de0b-4f06-bbb5-37522055a6e8" (UID: "184b5d2e-de0b-4f06-bbb5-37522055a6e8"). InnerVolumeSpecName "kube-api-access-4d4gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:06.765601 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.765502 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d4gl\" (UniqueName: \"kubernetes.io/projected/184b5d2e-de0b-4f06-bbb5-37522055a6e8-kube-api-access-4d4gl\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:06.765601 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:06.765540 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184b5d2e-de0b-4f06-bbb5-37522055a6e8-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:07.099716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.099576 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7"] Oct 30 12:14:07.100192 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.100169 4468 cpu_manager.go:414] "containerMap: removing container" podUID="184b5d2e-de0b-4f06-bbb5-37522055a6e8" containerName="payload-pre-processing" Oct 30 12:14:07.100192 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.100189 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184b5d2e-de0b-4f06-bbb5-37522055a6e8" containerName="payload-pre-processing" Oct 30 12:14:07.100324 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.100206 4468 cpu_manager.go:414] "containerMap: removing container" podUID="1be0127f-6554-4c65-be90-db22d98a6071" containerName="payload-processing" Oct 30 12:14:07.100324 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.100211 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be0127f-6554-4c65-be90-db22d98a6071" containerName="payload-processing" Oct 30 12:14:07.100324 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.100219 4468 cpu_manager.go:414] "containerMap: removing container" podUID="cc0ce46a-bfff-4509-97ae-58fca797cc56" containerName="payload-pre-processing" Oct 30 12:14:07.100324 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.100224 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0ce46a-bfff-4509-97ae-58fca797cc56" containerName="payload-pre-processing" Oct 30 12:14:07.100324 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.100255 4468 cpu_manager.go:414] "containerMap: removing container" podUID="20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb" containerName="payload-processing" Oct 30 12:14:07.100324 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.100261 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb" containerName="payload-processing" Oct 30 12:14:07.100324 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.100268 4468 cpu_manager.go:414] "containerMap: removing container" podUID="dac85f8a-9ce0-449f-b7a3-277c53941fd0" containerName="payload-pre-processing" Oct 30 12:14:07.100324 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.100273 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac85f8a-9ce0-449f-b7a3-277c53941fd0" containerName="payload-pre-processing" Oct 30 12:14:07.100324 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.100286 4468 cpu_manager.go:414] "containerMap: removing container" podUID="f581c0ea-a55a-4a4a-b892-45c8759ba074" containerName="payload-processing" Oct 30 12:14:07.100324 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.100290 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f581c0ea-a55a-4a4a-b892-45c8759ba074" containerName="payload-processing" Oct 30 12:14:07.100609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.100371 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="20aa3ee4-4dc0-4c81-93df-04db6ceaa7fb" containerName="payload-processing" Oct 30 12:14:07.100609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.100380 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac85f8a-9ce0-449f-b7a3-277c53941fd0" containerName="payload-pre-processing" Oct 30 12:14:07.100609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.100386 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f581c0ea-a55a-4a4a-b892-45c8759ba074" containerName="payload-processing" Oct 30 12:14:07.100609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.100395 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="184b5d2e-de0b-4f06-bbb5-37522055a6e8" containerName="payload-pre-processing" Oct 30 12:14:07.100609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.100401 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be0127f-6554-4c65-be90-db22d98a6071" containerName="payload-processing" Oct 30 12:14:07.100609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.100407 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0ce46a-bfff-4509-97ae-58fca797cc56" containerName="payload-pre-processing" Oct 30 12:14:07.102625 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.102604 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" Oct 30 12:14:07.137133 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.133880 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-6a7148-dockercfg-x6mql\"" Oct 30 12:14:07.139128 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.137922 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7"] Oct 30 12:14:07.173617 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.173198 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br9v8\" (UniqueName: \"kubernetes.io/projected/c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7-kube-api-access-br9v8\") pod \"payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7\" (UID: \"c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" Oct 30 12:14:07.173617 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.173335 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7\" (UID: \"c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" Oct 30 12:14:07.230256 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.210393 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-6a7148\"" Oct 30 12:14:07.275119 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.275043 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7\" (UID: \"c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" Oct 30 12:14:07.275275 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.275179 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br9v8\" (UniqueName: \"kubernetes.io/projected/c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7-kube-api-access-br9v8\") pod \"payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7\" (UID: \"c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" Oct 30 12:14:07.275793 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.275769 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7\" (UID: \"c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" Oct 30 12:14:07.332930 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.332825 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br9v8\" (UniqueName: \"kubernetes.io/projected/c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7-kube-api-access-br9v8\") pod \"payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7\" (UID: \"c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" Oct 30 12:14:07.421350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.421313 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn"] Oct 30 12:14:07.421905 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.421859 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" podUID="d5e74d16-3dde-444b-8f10-769314d02026" containerName="istio-proxy" containerID="cri-o://4e3b062f132b0f27398ad9fb98648ed43cebb0411e78434444910df4fdf4427f" gracePeriod=30 Oct 30 12:14:07.440663 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.428036 4468 patch_prober.go:29] interesting pod/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 12:14:07.440663 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.428119 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" podUID="f1338306-3768-445a-a6eb-57eafbbeeed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 12:14:07.472919 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.463348 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" Oct 30 12:14:07.480796 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.479701 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t" event={"ID":"1be0127f-6554-4c65-be90-db22d98a6071","Type":"ContainerDied","Data":"65c10e1600bac01b20d7bbabb06a6c133a07127199a22d9af287fd8278341c5f"} Oct 30 12:14:07.480796 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.479777 4468 scope.go:122] "RemoveContainer" containerID="039b2dcfd1e42bc947771ea02827bc507c472288f973ff2877de7ba2de5cf018" Oct 30 12:14:07.480796 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.479938 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t" Oct 30 12:14:07.484911 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.484126 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p" event={"ID":"184b5d2e-de0b-4f06-bbb5-37522055a6e8","Type":"ContainerDied","Data":"2ca3e4f48110ddca087fa10cf0bbd2ea9120e3e2bec2ddef2e246ba82f39e2e4"} Oct 30 12:14:07.484911 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.484250 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p" Oct 30 12:14:07.589735 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.589704 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p"] Oct 30 12:14:07.594466 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.594343 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-80143318-7d5b964bc9-jds2p"] Oct 30 12:14:07.602876 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.602852 4468 scope.go:122] "RemoveContainer" containerID="b127c61e210cc540d32191a6d5383eb78aea753f7ca44abd3335c5d0f3488a2b" Oct 30 12:14:07.631175 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.631136 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t"] Oct 30 12:14:07.634307 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.634262 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-80143318-7ccb5f4d78-bsb8t"] Oct 30 12:14:07.958475 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:07.958445 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7"] Oct 30 12:14:07.959063 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:14:07.959024 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d0e8b8_6a27_4d23_a6be_f1b8421a55c7.slice/crio-ac74747cb26c88fc9715a2ea61f2f5bc8fd7532012f3afbb9906066c5e138d68 WatchSource:0}: Error finding container ac74747cb26c88fc9715a2ea61f2f5bc8fd7532012f3afbb9906066c5e138d68: Status 404 returned error can't find the container with id ac74747cb26c88fc9715a2ea61f2f5bc8fd7532012f3afbb9906066c5e138d68 Oct 30 12:14:08.263584 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:08.263487 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27"] Oct 30 12:14:08.302102 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:08.302053 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2"] Oct 30 12:14:08.433103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:08.423637 4468 patch_prober.go:29] interesting pod/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 12:14:08.434485 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:08.434432 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" podUID="f1338306-3768-445a-a6eb-57eafbbeeed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 12:14:08.494810 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:08.494777 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" event={"ID":"c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7","Type":"ContainerStarted","Data":"6a67241a5ad94f978767b599658d275d86c4b62389f96fd05d1c8461e3d90cba"} Oct 30 12:14:08.496285 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:08.496251 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" event={"ID":"c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7","Type":"ContainerStarted","Data":"ac74747cb26c88fc9715a2ea61f2f5bc8fd7532012f3afbb9906066c5e138d68"} Oct 30 12:14:08.496446 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:08.496316 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" Oct 30 12:14:08.544821 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:08.544684 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" podStartSLOduration=1.544669652 podStartE2EDuration="1.544669652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:08.543452053 +0000 UTC m=+1199.577590450" watchObservedRunningTime="2026-10-30 12:14:08.544669652 +0000 UTC m=+1199.578808048" Oct 30 12:14:08.660838 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:08.660785 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf" podUID="4e4971c4-9bf1-4140-a050-2d6b84805b2d" containerName="payload-processing" containerID="cri-o://7bd9b950cb190bfeaaf1dece41bfb6c2d6fd43a80530cd946a4ecc881a6ec02d" gracePeriod=25 Oct 30 12:14:09.060641 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.059778 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf" Oct 30 12:14:09.089252 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.089218 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184b5d2e-de0b-4f06-bbb5-37522055a6e8" path="/var/lib/kubelet/pods/184b5d2e-de0b-4f06-bbb5-37522055a6e8/volumes" Oct 30 12:14:09.089614 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.089601 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1be0127f-6554-4c65-be90-db22d98a6071" path="/var/lib/kubelet/pods/1be0127f-6554-4c65-be90-db22d98a6071/volumes" Oct 30 12:14:09.147906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.147812 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4971c4-9bf1-4140-a050-2d6b84805b2d-kube-api-access-j7x5f\" (UniqueName: \"kubernetes.io/projected/4e4971c4-9bf1-4140-a050-2d6b84805b2d-kube-api-access-j7x5f\") pod \"4e4971c4-9bf1-4140-a050-2d6b84805b2d\" (UID: \"4e4971c4-9bf1-4140-a050-2d6b84805b2d\") " Oct 30 12:14:09.148193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.148169 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4971c4-9bf1-4140-a050-2d6b84805b2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4971c4-9bf1-4140-a050-2d6b84805b2d-plugins-config-volume\") pod \"4e4971c4-9bf1-4140-a050-2d6b84805b2d\" (UID: \"4e4971c4-9bf1-4140-a050-2d6b84805b2d\") " Oct 30 12:14:09.149183 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.149149 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4971c4-9bf1-4140-a050-2d6b84805b2d-plugins-config-volume" pod "4e4971c4-9bf1-4140-a050-2d6b84805b2d" (UID: "4e4971c4-9bf1-4140-a050-2d6b84805b2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:14:09.160309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.160270 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4971c4-9bf1-4140-a050-2d6b84805b2d-kube-api-access-j7x5f" pod "4e4971c4-9bf1-4140-a050-2d6b84805b2d" (UID: "4e4971c4-9bf1-4140-a050-2d6b84805b2d"). InnerVolumeSpecName "kube-api-access-j7x5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:09.249761 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.249741 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7x5f\" (UniqueName: \"kubernetes.io/projected/4e4971c4-9bf1-4140-a050-2d6b84805b2d-kube-api-access-j7x5f\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:09.249877 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.249867 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4971c4-9bf1-4140-a050-2d6b84805b2d-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:09.432796 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.431146 4468 patch_prober.go:29] interesting pod/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 12:14:09.432796 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.431206 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" podUID="f1338306-3768-445a-a6eb-57eafbbeeed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 12:14:09.521095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.503176 4468 generic.go:356] "Generic (PLEG): container finished" podID="4e4971c4-9bf1-4140-a050-2d6b84805b2d" containerID="7bd9b950cb190bfeaaf1dece41bfb6c2d6fd43a80530cd946a4ecc881a6ec02d" exitCode=0 Oct 30 12:14:09.521095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.504018 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf" Oct 30 12:14:09.521095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.516408 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf" event={"ID":"4e4971c4-9bf1-4140-a050-2d6b84805b2d","Type":"ContainerDied","Data":"7bd9b950cb190bfeaaf1dece41bfb6c2d6fd43a80530cd946a4ecc881a6ec02d"} Oct 30 12:14:09.521095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.516487 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf" event={"ID":"4e4971c4-9bf1-4140-a050-2d6b84805b2d","Type":"ContainerDied","Data":"aeefe7d42e58d2d66e8386c22ab59ba8b1ab752b07f4c9defdf54fd3bcfe6295"} Oct 30 12:14:09.521095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.516527 4468 scope.go:122] "RemoveContainer" containerID="7bd9b950cb190bfeaaf1dece41bfb6c2d6fd43a80530cd946a4ecc881a6ec02d" Oct 30 12:14:09.543740 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.543718 4468 scope.go:122] "RemoveContainer" containerID="7bd9b950cb190bfeaaf1dece41bfb6c2d6fd43a80530cd946a4ecc881a6ec02d" Oct 30 12:14:09.544146 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:14:09.544122 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd9b950cb190bfeaaf1dece41bfb6c2d6fd43a80530cd946a4ecc881a6ec02d\": container with ID starting with 7bd9b950cb190bfeaaf1dece41bfb6c2d6fd43a80530cd946a4ecc881a6ec02d not found: ID does not exist" containerID="7bd9b950cb190bfeaaf1dece41bfb6c2d6fd43a80530cd946a4ecc881a6ec02d" Oct 30 12:14:09.544199 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.544158 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd9b950cb190bfeaaf1dece41bfb6c2d6fd43a80530cd946a4ecc881a6ec02d"} err="failed to get container status \"7bd9b950cb190bfeaaf1dece41bfb6c2d6fd43a80530cd946a4ecc881a6ec02d\": rpc error: code = NotFound desc = could not find container \"7bd9b950cb190bfeaaf1dece41bfb6c2d6fd43a80530cd946a4ecc881a6ec02d\": container with ID starting with 7bd9b950cb190bfeaaf1dece41bfb6c2d6fd43a80530cd946a4ecc881a6ec02d not found: ID does not exist" Oct 30 12:14:09.552982 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.552950 4468 patch_prober.go:29] interesting pod/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 12:14:09.553180 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.553020 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" podUID="2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 12:14:09.578478 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.578243 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf"] Oct 30 12:14:09.580466 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.580441 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e636da41-7f85998d6-m5kxf"] Oct 30 12:14:09.758802 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.758768 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:14:09.852990 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.852955 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w"] Oct 30 12:14:09.853510 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.853491 4468 cpu_manager.go:414] "containerMap: removing container" podUID="2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" containerName="istio-proxy" Oct 30 12:14:09.853608 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.853512 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" containerName="istio-proxy" Oct 30 12:14:09.853608 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.853531 4468 cpu_manager.go:414] "containerMap: removing container" podUID="2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" containerName="istio-proxy" Oct 30 12:14:09.853608 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.853538 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" containerName="istio-proxy" Oct 30 12:14:09.853608 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.853563 4468 cpu_manager.go:414] "containerMap: removing container" podUID="4e4971c4-9bf1-4140-a050-2d6b84805b2d" containerName="payload-processing" Oct 30 12:14:09.853608 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.853571 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4971c4-9bf1-4140-a050-2d6b84805b2d" containerName="payload-processing" Oct 30 12:14:09.853849 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.853701 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" containerName="istio-proxy" Oct 30 12:14:09.853849 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.853718 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4971c4-9bf1-4140-a050-2d6b84805b2d" containerName="payload-processing" Oct 30 12:14:09.855457 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.855428 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-kube-api-access-gppgm\" (UniqueName: \"kubernetes.io/projected/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-kube-api-access-gppgm\") pod \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " Oct 30 12:14:09.855724 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.855702 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-workload-certs\") pod \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " Oct 30 12:14:09.856114 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.856057 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-data\") pod \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " Oct 30 12:14:09.856226 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.856204 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-workload-certs" pod "2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" (UID: "2eef0ee0-246f-4a05-9ba0-8b0746f23c2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:09.856454 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.856395 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-data" pod "2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" (UID: "2eef0ee0-246f-4a05-9ba0-8b0746f23c2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:09.856454 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.856418 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-credential-socket\") pod \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " Oct 30 12:14:09.856629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.856474 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-token\" (UniqueName: \"kubernetes.io/projected/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-token\") pod \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " Oct 30 12:14:09.856629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.856520 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-envoy\") pod \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " Oct 30 12:14:09.856629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.856545 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:09.856629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.856571 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istiod-ca-cert\") pod \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " Oct 30 12:14:09.856629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.856618 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-podinfo\") pod \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " Oct 30 12:14:09.856875 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.856650 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-workload-socket\") pod \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\" (UID: \"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b\") " Oct 30 12:14:09.857592 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.857032 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:09.857592 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.857055 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:09.857592 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.857267 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-workload-socket" pod "2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" (UID: "2eef0ee0-246f-4a05-9ba0-8b0746f23c2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:09.857592 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.857558 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istiod-ca-cert" pod "2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" (UID: "2eef0ee0-246f-4a05-9ba0-8b0746f23c2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:14:09.858361 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.858324 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-credential-socket" pod "2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" (UID: "2eef0ee0-246f-4a05-9ba0-8b0746f23c2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:09.862840 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.862780 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-podinfo" pod "2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" (UID: "2eef0ee0-246f-4a05-9ba0-8b0746f23c2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:14:09.862840 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.862791 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-envoy" pod "2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" (UID: "2eef0ee0-246f-4a05-9ba0-8b0746f23c2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:09.862840 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.862808 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-kube-api-access-gppgm" pod "2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" (UID: "2eef0ee0-246f-4a05-9ba0-8b0746f23c2b"). InnerVolumeSpecName "kube-api-access-gppgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:09.863265 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.863244 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-token" pod "2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" (UID: "2eef0ee0-246f-4a05-9ba0-8b0746f23c2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:09.879179 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.876651 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f920f7-openshift-default-dockercfg-d89vp\"" Oct 30 12:14:09.881460 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.881429 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w"] Oct 30 12:14:09.957830 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.957741 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:09.957830 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.957787 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7p57\" (UniqueName: \"kubernetes.io/projected/189793bd-8385-4741-85aa-3d89fc9a1d30-kube-api-access-n7p57\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:09.957830 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.957809 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:09.958116 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.957964 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:09.958116 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.958000 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189793bd-8385-4741-85aa-3d89fc9a1d30-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:09.958116 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.958076 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:09.958230 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.958142 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:09.958230 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.958194 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:09.958230 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.958227 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:09.958349 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.958338 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:09.958389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.958351 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:09.958389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.958360 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:09.958389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.958368 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:09.958389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.958376 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:09.958389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.958384 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:09.958532 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:09.958392 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gppgm\" (UniqueName: \"kubernetes.io/projected/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b-kube-api-access-gppgm\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:10.070287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.059277 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:10.070287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.059337 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:10.070287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.059386 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:10.070287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.059499 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:10.070287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.059537 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7p57\" (UniqueName: \"kubernetes.io/projected/189793bd-8385-4741-85aa-3d89fc9a1d30-kube-api-access-n7p57\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:10.070287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.059584 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:10.070287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.059623 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:10.070287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.059662 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189793bd-8385-4741-85aa-3d89fc9a1d30-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:10.070287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.059696 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:10.070287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.060218 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:10.070287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.060919 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:10.070287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.061195 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:10.070287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.064127 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:10.070287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.067806 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:10.070287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.068084 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:10.070287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.068682 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189793bd-8385-4741-85aa-3d89fc9a1d30-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:10.084153 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.084123 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:10.089405 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.089372 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7p57\" (UniqueName: \"kubernetes.io/projected/189793bd-8385-4741-85aa-3d89fc9a1d30-kube-api-access-n7p57\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:10.189646 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.189612 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:10.415348 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.414819 4468 patch_prober.go:29] interesting pod/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 12:14:10.415348 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.414883 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" podUID="f1338306-3768-445a-a6eb-57eafbbeeed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 12:14:10.511523 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.511485 4468 generic.go:356] "Generic (PLEG): container finished" podID="2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" containerID="569fb20fcf136fac6307d7aafee9099a009a851dc62a4f8385b9c2d22f58c4fd" exitCode=0 Oct 30 12:14:10.512015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.511552 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" Oct 30 12:14:10.512015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.511570 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" event={"ID":"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b","Type":"ContainerDied","Data":"569fb20fcf136fac6307d7aafee9099a009a851dc62a4f8385b9c2d22f58c4fd"} Oct 30 12:14:10.512015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.511633 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv" event={"ID":"2eef0ee0-246f-4a05-9ba0-8b0746f23c2b","Type":"ContainerDied","Data":"6f835e2382055c950ca18a14cccbbc950aea0ee1db20d875568afc365921650e"} Oct 30 12:14:10.512015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.511666 4468 scope.go:122] "RemoveContainer" containerID="569fb20fcf136fac6307d7aafee9099a009a851dc62a4f8385b9c2d22f58c4fd" Oct 30 12:14:10.526476 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.526453 4468 scope.go:122] "RemoveContainer" containerID="544320268b1126839dc8d7b3f0e7037a30d6171d202513f4bd6842b2a280f3d3" Oct 30 12:14:10.538026 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.537996 4468 scope.go:122] "RemoveContainer" containerID="569fb20fcf136fac6307d7aafee9099a009a851dc62a4f8385b9c2d22f58c4fd" Oct 30 12:14:10.538530 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:14:10.538501 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"569fb20fcf136fac6307d7aafee9099a009a851dc62a4f8385b9c2d22f58c4fd\": container with ID starting with 569fb20fcf136fac6307d7aafee9099a009a851dc62a4f8385b9c2d22f58c4fd not found: ID does not exist" containerID="569fb20fcf136fac6307d7aafee9099a009a851dc62a4f8385b9c2d22f58c4fd" Oct 30 12:14:10.538652 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.538540 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"569fb20fcf136fac6307d7aafee9099a009a851dc62a4f8385b9c2d22f58c4fd"} err="failed to get container status \"569fb20fcf136fac6307d7aafee9099a009a851dc62a4f8385b9c2d22f58c4fd\": rpc error: code = NotFound desc = could not find container \"569fb20fcf136fac6307d7aafee9099a009a851dc62a4f8385b9c2d22f58c4fd\": container with ID starting with 569fb20fcf136fac6307d7aafee9099a009a851dc62a4f8385b9c2d22f58c4fd not found: ID does not exist" Oct 30 12:14:10.538652 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.538566 4468 scope.go:122] "RemoveContainer" containerID="544320268b1126839dc8d7b3f0e7037a30d6171d202513f4bd6842b2a280f3d3" Oct 30 12:14:10.538920 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:14:10.538902 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"544320268b1126839dc8d7b3f0e7037a30d6171d202513f4bd6842b2a280f3d3\": container with ID starting with 544320268b1126839dc8d7b3f0e7037a30d6171d202513f4bd6842b2a280f3d3 not found: ID does not exist" containerID="544320268b1126839dc8d7b3f0e7037a30d6171d202513f4bd6842b2a280f3d3" Oct 30 12:14:10.539011 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.538924 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"544320268b1126839dc8d7b3f0e7037a30d6171d202513f4bd6842b2a280f3d3"} err="failed to get container status \"544320268b1126839dc8d7b3f0e7037a30d6171d202513f4bd6842b2a280f3d3\": rpc error: code = NotFound desc = could not find container \"544320268b1126839dc8d7b3f0e7037a30d6171d202513f4bd6842b2a280f3d3\": container with ID starting with 544320268b1126839dc8d7b3f0e7037a30d6171d202513f4bd6842b2a280f3d3 not found: ID does not exist" Oct 30 12:14:10.592100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.591944 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w"] Oct 30 12:14:10.594300 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.594224 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:14:10.594402 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.594356 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:14:10.594496 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.594440 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:14:10.621102 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.621056 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv"] Oct 30 12:14:10.637759 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:10.637724 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1c85bd66-openshift-default-56c4c985d7-gklqv"] Oct 30 12:14:11.089725 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.089694 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" path="/var/lib/kubelet/pods/2eef0ee0-246f-4a05-9ba0-8b0746f23c2b/volumes" Oct 30 12:14:11.090245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.090228 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4971c4-9bf1-4140-a050-2d6b84805b2d" path="/var/lib/kubelet/pods/4e4971c4-9bf1-4140-a050-2d6b84805b2d/volumes" Oct 30 12:14:11.308755 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.308722 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2"] Oct 30 12:14:11.309277 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.309263 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eef0ee0-246f-4a05-9ba0-8b0746f23c2b" containerName="istio-proxy" Oct 30 12:14:11.312062 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.312044 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" Oct 30 12:14:11.336767 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.336727 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2"] Oct 30 12:14:11.337149 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.337120 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-512ec5\"" Oct 30 12:14:11.337917 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.337895 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-512ec5-dockercfg-sqcrs\"" Oct 30 12:14:11.436429 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.433637 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt"] Oct 30 12:14:11.444167 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.444140 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:14:11.444365 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.444353 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt"] Oct 30 12:14:11.444782 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.444761 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" Oct 30 12:14:11.445803 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.445715 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:14:11.449334 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.449312 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:14:11.527477 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.527439 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltgvk\" (UniqueName: \"kubernetes.io/projected/ca8bf37f-b34e-42b2-810e-96d117d30e13-kube-api-access-ltgvk\") pod \"payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2\" (UID: \"ca8bf37f-b34e-42b2-810e-96d117d30e13\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" Oct 30 12:14:11.527974 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.527517 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8bf37f-b34e-42b2-810e-96d117d30e13-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2\" (UID: \"ca8bf37f-b34e-42b2-810e-96d117d30e13\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" Oct 30 12:14:11.528581 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.528548 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" event={"ID":"189793bd-8385-4741-85aa-3d89fc9a1d30","Type":"ContainerStarted","Data":"8bc6e9749f1b5d86d39134cea2c68e807cab5bcd4fbfeecd1d587e2919976070"} Oct 30 12:14:11.528697 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.528611 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" event={"ID":"189793bd-8385-4741-85aa-3d89fc9a1d30","Type":"ContainerStarted","Data":"7b8d1aabfd2b83485adfe2fd5b34676cad3cbda7c3c97e12b7fe601ccc68fad8"} Oct 30 12:14:11.579580 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.579517 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" podStartSLOduration=2.579497494 podStartE2EDuration="2.579497494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:11.578740767 +0000 UTC m=+1202.612879174" watchObservedRunningTime="2026-10-30 12:14:11.579497494 +0000 UTC m=+1202.613635892" Oct 30 12:14:11.628716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.628662 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltgvk\" (UniqueName: \"kubernetes.io/projected/ca8bf37f-b34e-42b2-810e-96d117d30e13-kube-api-access-ltgvk\") pod \"payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2\" (UID: \"ca8bf37f-b34e-42b2-810e-96d117d30e13\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" Oct 30 12:14:11.628933 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.628723 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8bf37f-b34e-42b2-810e-96d117d30e13-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2\" (UID: \"ca8bf37f-b34e-42b2-810e-96d117d30e13\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" Oct 30 12:14:11.628933 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.628917 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe0584e-df4b-4393-9313-9f347cc38767-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt\" (UID: \"fbe0584e-df4b-4393-9313-9f347cc38767\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" Oct 30 12:14:11.629100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.629048 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9t2s\" (UniqueName: \"kubernetes.io/projected/fbe0584e-df4b-4393-9313-9f347cc38767-kube-api-access-z9t2s\") pod \"payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt\" (UID: \"fbe0584e-df4b-4393-9313-9f347cc38767\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" Oct 30 12:14:11.629757 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.629732 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8bf37f-b34e-42b2-810e-96d117d30e13-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2\" (UID: \"ca8bf37f-b34e-42b2-810e-96d117d30e13\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" Oct 30 12:14:11.652716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.652678 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltgvk\" (UniqueName: \"kubernetes.io/projected/ca8bf37f-b34e-42b2-810e-96d117d30e13-kube-api-access-ltgvk\") pod \"payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2\" (UID: \"ca8bf37f-b34e-42b2-810e-96d117d30e13\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" Oct 30 12:14:11.731198 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.730817 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9t2s\" (UniqueName: \"kubernetes.io/projected/fbe0584e-df4b-4393-9313-9f347cc38767-kube-api-access-z9t2s\") pod \"payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt\" (UID: \"fbe0584e-df4b-4393-9313-9f347cc38767\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" Oct 30 12:14:11.731198 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.731038 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe0584e-df4b-4393-9313-9f347cc38767-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt\" (UID: \"fbe0584e-df4b-4393-9313-9f347cc38767\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" Oct 30 12:14:11.754170 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.731999 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe0584e-df4b-4393-9313-9f347cc38767-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt\" (UID: \"fbe0584e-df4b-4393-9313-9f347cc38767\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" Oct 30 12:14:11.769224 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.759134 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9t2s\" (UniqueName: \"kubernetes.io/projected/fbe0584e-df4b-4393-9313-9f347cc38767-kube-api-access-z9t2s\") pod \"payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt\" (UID: \"fbe0584e-df4b-4393-9313-9f347cc38767\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" Oct 30 12:14:11.769224 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.763149 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" Oct 30 12:14:11.923898 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:11.923863 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" Oct 30 12:14:12.073002 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:12.072969 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt"] Oct 30 12:14:12.190210 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:12.190174 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:12.195925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:12.195898 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:12.226489 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:12.226463 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2"] Oct 30 12:14:12.537025 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:12.536910 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" event={"ID":"fbe0584e-df4b-4393-9313-9f347cc38767","Type":"ContainerStarted","Data":"c5e801734b99770487116591d475290cf177a00eb649137635f83bb1a7e59e9b"} Oct 30 12:14:12.537505 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:12.537122 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" Oct 30 12:14:12.537505 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:12.537159 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" event={"ID":"fbe0584e-df4b-4393-9313-9f347cc38767","Type":"ContainerStarted","Data":"80fa79dfd7a767a1dc38729319c44e7427f59bf6c3fd2319dfddbf45396e63db"} Oct 30 12:14:12.539457 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:12.539420 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" event={"ID":"ca8bf37f-b34e-42b2-810e-96d117d30e13","Type":"ContainerStarted","Data":"2f9ad94cab59ed1f933f7057334fad07db2582961b38701d2dff27b550930844"} Oct 30 12:14:12.539609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:12.539478 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" event={"ID":"ca8bf37f-b34e-42b2-810e-96d117d30e13","Type":"ContainerStarted","Data":"a427360fbb40a08453346472696a751bbd807756f29c2e379d912ab5b24604d3"} Oct 30 12:14:12.539975 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:12.539943 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:12.540583 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:12.540565 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:12.576304 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:12.576249 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" podStartSLOduration=1.576236866 podStartE2EDuration="1.576236866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:12.576132377 +0000 UTC m=+1203.610270770" watchObservedRunningTime="2026-10-30 12:14:12.576236866 +0000 UTC m=+1203.610375261" Oct 30 12:14:12.618384 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:12.618329 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" podStartSLOduration=1.6183141220000001 podStartE2EDuration="1.618314122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:12.616922401 +0000 UTC m=+1203.651060797" watchObservedRunningTime="2026-10-30 12:14:12.618314122 +0000 UTC m=+1203.652452517" Oct 30 12:14:12.693198 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:14:12.690509 4468 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e74d16_3dde_444b_8f10_769314d02026.slice/crio-77faf8b6462ec60d6f1d332a852791f223fc1e10f91fd0961864e3708fc03092\": RecentStats: unable to find data in memory cache]" Oct 30 12:14:12.867094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:12.855041 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:14:13.044265 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.044228 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-workload-socket\") pod \"d5e74d16-3dde-444b-8f10-769314d02026\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " Oct 30 12:14:13.044452 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.044334 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e74d16-3dde-444b-8f10-769314d02026-kube-api-access-7l5f8\" (UniqueName: \"kubernetes.io/projected/d5e74d16-3dde-444b-8f10-769314d02026-kube-api-access-7l5f8\") pod \"d5e74d16-3dde-444b-8f10-769314d02026\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " Oct 30 12:14:13.044452 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.044358 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-workload-certs\") pod \"d5e74d16-3dde-444b-8f10-769314d02026\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " Oct 30 12:14:13.044452 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.044391 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5e74d16-3dde-444b-8f10-769314d02026-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e74d16-3dde-444b-8f10-769314d02026-istio-podinfo\") pod \"d5e74d16-3dde-444b-8f10-769314d02026\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " Oct 30 12:14:13.044452 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.044412 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e74d16-3dde-444b-8f10-769314d02026-istio-token\" (UniqueName: \"kubernetes.io/projected/d5e74d16-3dde-444b-8f10-769314d02026-istio-token\") pod \"d5e74d16-3dde-444b-8f10-769314d02026\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " Oct 30 12:14:13.044452 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.044432 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-credential-socket\") pod \"d5e74d16-3dde-444b-8f10-769314d02026\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " Oct 30 12:14:13.044452 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.044449 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-istio-data\") pod \"d5e74d16-3dde-444b-8f10-769314d02026\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " Oct 30 12:14:13.044768 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.044488 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-istio-envoy\") pod \"d5e74d16-3dde-444b-8f10-769314d02026\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " Oct 30 12:14:13.044768 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.044527 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5e74d16-3dde-444b-8f10-769314d02026-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e74d16-3dde-444b-8f10-769314d02026-istiod-ca-cert\") pod \"d5e74d16-3dde-444b-8f10-769314d02026\" (UID: \"d5e74d16-3dde-444b-8f10-769314d02026\") " Oct 30 12:14:13.044768 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.044579 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-workload-socket" pod "d5e74d16-3dde-444b-8f10-769314d02026" (UID: "d5e74d16-3dde-444b-8f10-769314d02026"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:13.044768 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.044697 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-workload-certs" pod "d5e74d16-3dde-444b-8f10-769314d02026" (UID: "d5e74d16-3dde-444b-8f10-769314d02026"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:13.045000 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.044796 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-istio-data" pod "d5e74d16-3dde-444b-8f10-769314d02026" (UID: "d5e74d16-3dde-444b-8f10-769314d02026"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:13.045062 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.045015 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.045062 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.045034 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.045062 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.045048 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.045240 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.045054 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-credential-socket" pod "d5e74d16-3dde-444b-8f10-769314d02026" (UID: "d5e74d16-3dde-444b-8f10-769314d02026"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:13.045240 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.045082 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e74d16-3dde-444b-8f10-769314d02026-istiod-ca-cert" pod "d5e74d16-3dde-444b-8f10-769314d02026" (UID: "d5e74d16-3dde-444b-8f10-769314d02026"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:14:13.046626 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.046600 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e74d16-3dde-444b-8f10-769314d02026-istio-token" pod "d5e74d16-3dde-444b-8f10-769314d02026" (UID: "d5e74d16-3dde-444b-8f10-769314d02026"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:13.046749 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.046686 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5e74d16-3dde-444b-8f10-769314d02026-istio-podinfo" pod "d5e74d16-3dde-444b-8f10-769314d02026" (UID: "d5e74d16-3dde-444b-8f10-769314d02026"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:14:13.046749 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.046687 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e74d16-3dde-444b-8f10-769314d02026-kube-api-access-7l5f8" pod "d5e74d16-3dde-444b-8f10-769314d02026" (UID: "d5e74d16-3dde-444b-8f10-769314d02026"). InnerVolumeSpecName "kube-api-access-7l5f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:13.046826 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.046770 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-istio-envoy" pod "d5e74d16-3dde-444b-8f10-769314d02026" (UID: "d5e74d16-3dde-444b-8f10-769314d02026"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:13.146247 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.146206 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l5f8\" (UniqueName: \"kubernetes.io/projected/d5e74d16-3dde-444b-8f10-769314d02026-kube-api-access-7l5f8\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.146247 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.146240 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e74d16-3dde-444b-8f10-769314d02026-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.146247 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.146252 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5e74d16-3dde-444b-8f10-769314d02026-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.146495 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.146260 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.146495 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.146270 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e74d16-3dde-444b-8f10-769314d02026-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.146495 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.146281 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e74d16-3dde-444b-8f10-769314d02026-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.264534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.264489 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27" podUID="60230ddf-11ad-4c9d-b49e-9d2547dc929e" containerName="payload-processing" containerID="cri-o://e6534cde6cddf7149fca7024cb021cf9bd9e77567ec13507ee9811583ba18bac" gracePeriod=25 Oct 30 12:14:13.302796 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.302757 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2" podUID="28f409f1-8fa9-438b-b05e-128b59362c24" containerName="payload-pre-processing" containerID="cri-o://3c69b7daa2868ab216112bb2f0b89d4c0f17db8f62f1995310b30e8723311837" gracePeriod=25 Oct 30 12:14:13.547541 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.547511 4468 generic.go:356] "Generic (PLEG): container finished" podID="60230ddf-11ad-4c9d-b49e-9d2547dc929e" containerID="e6534cde6cddf7149fca7024cb021cf9bd9e77567ec13507ee9811583ba18bac" exitCode=0 Oct 30 12:14:13.547938 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.547584 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27" event={"ID":"60230ddf-11ad-4c9d-b49e-9d2547dc929e","Type":"ContainerDied","Data":"e6534cde6cddf7149fca7024cb021cf9bd9e77567ec13507ee9811583ba18bac"} Oct 30 12:14:13.547938 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.547646 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27" event={"ID":"60230ddf-11ad-4c9d-b49e-9d2547dc929e","Type":"ContainerDied","Data":"c6ee14ea062f8240dcf3819e4ac1d3a6b928b9cbdd8a2ac86e14810a07cbd3f9"} Oct 30 12:14:13.547938 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.547673 4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6ee14ea062f8240dcf3819e4ac1d3a6b928b9cbdd8a2ac86e14810a07cbd3f9" Oct 30 12:14:13.547938 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.547719 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27" Oct 30 12:14:13.550319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.550291 4468 generic.go:356] "Generic (PLEG): container finished" podID="d5e74d16-3dde-444b-8f10-769314d02026" containerID="4e3b062f132b0f27398ad9fb98648ed43cebb0411e78434444910df4fdf4427f" exitCode=0 Oct 30 12:14:13.550465 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.550319 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" event={"ID":"d5e74d16-3dde-444b-8f10-769314d02026","Type":"ContainerDied","Data":"4e3b062f132b0f27398ad9fb98648ed43cebb0411e78434444910df4fdf4427f"} Oct 30 12:14:13.550465 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.550353 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" Oct 30 12:14:13.550465 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.550371 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn" event={"ID":"d5e74d16-3dde-444b-8f10-769314d02026","Type":"ContainerDied","Data":"77faf8b6462ec60d6f1d332a852791f223fc1e10f91fd0961864e3708fc03092"} Oct 30 12:14:13.550465 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.550402 4468 scope.go:122] "RemoveContainer" containerID="4e3b062f132b0f27398ad9fb98648ed43cebb0411e78434444910df4fdf4427f" Oct 30 12:14:13.553222 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.553198 4468 generic.go:356] "Generic (PLEG): container finished" podID="28f409f1-8fa9-438b-b05e-128b59362c24" containerID="3c69b7daa2868ab216112bb2f0b89d4c0f17db8f62f1995310b30e8723311837" exitCode=0 Oct 30 12:14:13.553325 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.553227 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2" event={"ID":"28f409f1-8fa9-438b-b05e-128b59362c24","Type":"ContainerDied","Data":"3c69b7daa2868ab216112bb2f0b89d4c0f17db8f62f1995310b30e8723311837"} Oct 30 12:14:13.554242 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.554224 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" Oct 30 12:14:13.561555 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.561530 4468 scope.go:122] "RemoveContainer" containerID="0fc49bd91096e7afd1d01a24d6e748edc6b1419d2c8358803cb1ff7ca5458a01" Oct 30 12:14:13.571791 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.571764 4468 scope.go:122] "RemoveContainer" containerID="4e3b062f132b0f27398ad9fb98648ed43cebb0411e78434444910df4fdf4427f" Oct 30 12:14:13.572163 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:14:13.572142 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3b062f132b0f27398ad9fb98648ed43cebb0411e78434444910df4fdf4427f\": container with ID starting with 4e3b062f132b0f27398ad9fb98648ed43cebb0411e78434444910df4fdf4427f not found: ID does not exist" containerID="4e3b062f132b0f27398ad9fb98648ed43cebb0411e78434444910df4fdf4427f" Oct 30 12:14:13.572233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.572174 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3b062f132b0f27398ad9fb98648ed43cebb0411e78434444910df4fdf4427f"} err="failed to get container status \"4e3b062f132b0f27398ad9fb98648ed43cebb0411e78434444910df4fdf4427f\": rpc error: code = NotFound desc = could not find container \"4e3b062f132b0f27398ad9fb98648ed43cebb0411e78434444910df4fdf4427f\": container with ID starting with 4e3b062f132b0f27398ad9fb98648ed43cebb0411e78434444910df4fdf4427f not found: ID does not exist" Oct 30 12:14:13.572233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.572193 4468 scope.go:122] "RemoveContainer" containerID="0fc49bd91096e7afd1d01a24d6e748edc6b1419d2c8358803cb1ff7ca5458a01" Oct 30 12:14:13.572537 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:14:13.572516 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc49bd91096e7afd1d01a24d6e748edc6b1419d2c8358803cb1ff7ca5458a01\": container with ID starting with 0fc49bd91096e7afd1d01a24d6e748edc6b1419d2c8358803cb1ff7ca5458a01 not found: ID does not exist" containerID="0fc49bd91096e7afd1d01a24d6e748edc6b1419d2c8358803cb1ff7ca5458a01" Oct 30 12:14:13.572594 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.572549 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc49bd91096e7afd1d01a24d6e748edc6b1419d2c8358803cb1ff7ca5458a01"} err="failed to get container status \"0fc49bd91096e7afd1d01a24d6e748edc6b1419d2c8358803cb1ff7ca5458a01\": rpc error: code = NotFound desc = could not find container \"0fc49bd91096e7afd1d01a24d6e748edc6b1419d2c8358803cb1ff7ca5458a01\": container with ID starting with 0fc49bd91096e7afd1d01a24d6e748edc6b1419d2c8358803cb1ff7ca5458a01 not found: ID does not exist" Oct 30 12:14:13.589742 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.589705 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2" Oct 30 12:14:13.621594 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.621557 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn"] Oct 30 12:14:13.633017 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.632986 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-80143318-openshift-default-8bd7b54cc-xnnfn"] Oct 30 12:14:13.651259 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.651218 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60230ddf-11ad-4c9d-b49e-9d2547dc929e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60230ddf-11ad-4c9d-b49e-9d2547dc929e-plugins-config-volume\") pod \"60230ddf-11ad-4c9d-b49e-9d2547dc929e\" (UID: \"60230ddf-11ad-4c9d-b49e-9d2547dc929e\") " Oct 30 12:14:13.651641 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.651608 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60230ddf-11ad-4c9d-b49e-9d2547dc929e-plugins-config-volume" pod "60230ddf-11ad-4c9d-b49e-9d2547dc929e" (UID: "60230ddf-11ad-4c9d-b49e-9d2547dc929e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:14:13.651805 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.651790 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60230ddf-11ad-4c9d-b49e-9d2547dc929e-kube-api-access-vv6dw\" (UniqueName: \"kubernetes.io/projected/60230ddf-11ad-4c9d-b49e-9d2547dc929e-kube-api-access-vv6dw\") pod \"60230ddf-11ad-4c9d-b49e-9d2547dc929e\" (UID: \"60230ddf-11ad-4c9d-b49e-9d2547dc929e\") " Oct 30 12:14:13.652250 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.652224 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60230ddf-11ad-4c9d-b49e-9d2547dc929e-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.654001 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.653978 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60230ddf-11ad-4c9d-b49e-9d2547dc929e-kube-api-access-vv6dw" pod "60230ddf-11ad-4c9d-b49e-9d2547dc929e" (UID: "60230ddf-11ad-4c9d-b49e-9d2547dc929e"). InnerVolumeSpecName "kube-api-access-vv6dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:13.752925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.752886 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28f409f1-8fa9-438b-b05e-128b59362c24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f409f1-8fa9-438b-b05e-128b59362c24-plugins-config-volume\") pod \"28f409f1-8fa9-438b-b05e-128b59362c24\" (UID: \"28f409f1-8fa9-438b-b05e-128b59362c24\") " Oct 30 12:14:13.753172 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.752981 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f409f1-8fa9-438b-b05e-128b59362c24-kube-api-access-2mxlj\" (UniqueName: \"kubernetes.io/projected/28f409f1-8fa9-438b-b05e-128b59362c24-kube-api-access-2mxlj\") pod \"28f409f1-8fa9-438b-b05e-128b59362c24\" (UID: \"28f409f1-8fa9-438b-b05e-128b59362c24\") " Oct 30 12:14:13.753385 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.753360 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f409f1-8fa9-438b-b05e-128b59362c24-plugins-config-volume" pod "28f409f1-8fa9-438b-b05e-128b59362c24" (UID: "28f409f1-8fa9-438b-b05e-128b59362c24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:14:13.753430 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.753377 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv6dw\" (UniqueName: \"kubernetes.io/projected/60230ddf-11ad-4c9d-b49e-9d2547dc929e-kube-api-access-vv6dw\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.755157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.755129 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f409f1-8fa9-438b-b05e-128b59362c24-kube-api-access-2mxlj" pod "28f409f1-8fa9-438b-b05e-128b59362c24" (UID: "28f409f1-8fa9-438b-b05e-128b59362c24"). InnerVolumeSpecName "kube-api-access-2mxlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:13.854670 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.854633 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mxlj\" (UniqueName: \"kubernetes.io/projected/28f409f1-8fa9-438b-b05e-128b59362c24-kube-api-access-2mxlj\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.854670 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:13.854668 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f409f1-8fa9-438b-b05e-128b59362c24-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:14.181390 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.181352 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn"] Oct 30 12:14:14.181973 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.181845 4468 cpu_manager.go:414] "containerMap: removing container" podUID="28f409f1-8fa9-438b-b05e-128b59362c24" containerName="payload-pre-processing" Oct 30 12:14:14.181973 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.181864 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f409f1-8fa9-438b-b05e-128b59362c24" containerName="payload-pre-processing" Oct 30 12:14:14.181973 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.181878 4468 cpu_manager.go:414] "containerMap: removing container" podUID="d5e74d16-3dde-444b-8f10-769314d02026" containerName="istio-proxy" Oct 30 12:14:14.181973 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.181884 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e74d16-3dde-444b-8f10-769314d02026" containerName="istio-proxy" Oct 30 12:14:14.181973 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.181898 4468 cpu_manager.go:414] "containerMap: removing container" podUID="60230ddf-11ad-4c9d-b49e-9d2547dc929e" containerName="payload-processing" Oct 30 12:14:14.181973 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.181906 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60230ddf-11ad-4c9d-b49e-9d2547dc929e" containerName="payload-processing" Oct 30 12:14:14.181973 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.181927 4468 cpu_manager.go:414] "containerMap: removing container" podUID="d5e74d16-3dde-444b-8f10-769314d02026" containerName="istio-proxy" Oct 30 12:14:14.181973 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.181932 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e74d16-3dde-444b-8f10-769314d02026" containerName="istio-proxy" Oct 30 12:14:14.182418 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.182003 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="60230ddf-11ad-4c9d-b49e-9d2547dc929e" containerName="payload-processing" Oct 30 12:14:14.182418 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.182014 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e74d16-3dde-444b-8f10-769314d02026" containerName="istio-proxy" Oct 30 12:14:14.182418 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.182023 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f409f1-8fa9-438b-b05e-128b59362c24" containerName="payload-pre-processing" Oct 30 12:14:14.184194 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.184174 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" Oct 30 12:14:14.203931 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.203896 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn"] Oct 30 12:14:14.206414 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.206383 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-47fd99\"" Oct 30 12:14:14.218602 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.218568 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s"] Oct 30 12:14:14.219133 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.219118 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e74d16-3dde-444b-8f10-769314d02026" containerName="istio-proxy" Oct 30 12:14:14.221168 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.221151 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" Oct 30 12:14:14.249340 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.249301 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s"] Oct 30 12:14:14.254026 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.253994 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-47fd99-dockercfg-gcppk\"" Oct 30 12:14:14.258256 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.258222 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4572318d-2ff5-43d0-9071-faeb6e94c178-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn\" (UID: \"4572318d-2ff5-43d0-9071-faeb6e94c178\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" Oct 30 12:14:14.258412 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.258293 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127e9882-c84b-4162-9565-834a8126f9e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s\" (UID: \"127e9882-c84b-4162-9565-834a8126f9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" Oct 30 12:14:14.258476 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.258453 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwwnk\" (UniqueName: \"kubernetes.io/projected/4572318d-2ff5-43d0-9071-faeb6e94c178-kube-api-access-xwwnk\") pod \"payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn\" (UID: \"4572318d-2ff5-43d0-9071-faeb6e94c178\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" Oct 30 12:14:14.258585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.258568 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6mzr\" (UniqueName: \"kubernetes.io/projected/127e9882-c84b-4162-9565-834a8126f9e3-kube-api-access-r6mzr\") pod \"payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s\" (UID: \"127e9882-c84b-4162-9565-834a8126f9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" Oct 30 12:14:14.360112 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.359988 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4572318d-2ff5-43d0-9071-faeb6e94c178-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn\" (UID: \"4572318d-2ff5-43d0-9071-faeb6e94c178\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" Oct 30 12:14:14.360347 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.360291 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127e9882-c84b-4162-9565-834a8126f9e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s\" (UID: \"127e9882-c84b-4162-9565-834a8126f9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" Oct 30 12:14:14.372161 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.372126 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4572318d-2ff5-43d0-9071-faeb6e94c178-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn\" (UID: \"4572318d-2ff5-43d0-9071-faeb6e94c178\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" Oct 30 12:14:14.372161 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.372126 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127e9882-c84b-4162-9565-834a8126f9e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s\" (UID: \"127e9882-c84b-4162-9565-834a8126f9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" Oct 30 12:14:14.372404 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.372364 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwwnk\" (UniqueName: \"kubernetes.io/projected/4572318d-2ff5-43d0-9071-faeb6e94c178-kube-api-access-xwwnk\") pod \"payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn\" (UID: \"4572318d-2ff5-43d0-9071-faeb6e94c178\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" Oct 30 12:14:14.372878 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.372848 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6mzr\" (UniqueName: \"kubernetes.io/projected/127e9882-c84b-4162-9565-834a8126f9e3-kube-api-access-r6mzr\") pod \"payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s\" (UID: \"127e9882-c84b-4162-9565-834a8126f9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" Oct 30 12:14:14.416143 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.415996 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6mzr\" (UniqueName: \"kubernetes.io/projected/127e9882-c84b-4162-9565-834a8126f9e3-kube-api-access-r6mzr\") pod \"payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s\" (UID: \"127e9882-c84b-4162-9565-834a8126f9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" Oct 30 12:14:14.416360 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.416150 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwwnk\" (UniqueName: \"kubernetes.io/projected/4572318d-2ff5-43d0-9071-faeb6e94c178-kube-api-access-xwwnk\") pod \"payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn\" (UID: \"4572318d-2ff5-43d0-9071-faeb6e94c178\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" Oct 30 12:14:14.503831 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.503721 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" Oct 30 12:14:14.537689 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.537655 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" Oct 30 12:14:14.569715 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.565236 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2" Oct 30 12:14:14.569715 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.567281 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27" Oct 30 12:14:14.569715 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.568206 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2" event={"ID":"28f409f1-8fa9-438b-b05e-128b59362c24","Type":"ContainerDied","Data":"5c0de58ca642ab415520317f719353f26d402be1a782f008f85fca378e5e6740"} Oct 30 12:14:14.569715 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.568278 4468 scope.go:122] "RemoveContainer" containerID="3c69b7daa2868ab216112bb2f0b89d4c0f17db8f62f1995310b30e8723311837" Oct 30 12:14:14.653242 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.653204 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2"] Oct 30 12:14:14.661273 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.661191 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-dfd96f0e-56687895f5-49tv2"] Oct 30 12:14:14.693621 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.693585 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27"] Oct 30 12:14:14.697577 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.697551 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-dfd96f0e-7c56966f44-6gc27"] Oct 30 12:14:14.808662 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.808631 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn"] Oct 30 12:14:14.808951 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:14:14.808931 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4572318d_2ff5_43d0_9071_faeb6e94c178.slice/crio-87ff20470cb3380054592a2b8b9a4e81e7b7acdb161c3b4bc1e016f132c7175b WatchSource:0}: Error finding container 87ff20470cb3380054592a2b8b9a4e81e7b7acdb161c3b4bc1e016f132c7175b: Status 404 returned error can't find the container with id 87ff20470cb3380054592a2b8b9a4e81e7b7acdb161c3b4bc1e016f132c7175b Oct 30 12:14:14.857141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:14.857113 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s"] Oct 30 12:14:14.857963 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:14:14.857936 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod127e9882_c84b_4162_9565_834a8126f9e3.slice/crio-f20153659601b8a9f0d332953bcf9e7184688985211d041e4fdb8c09535cfbb3 WatchSource:0}: Error finding container f20153659601b8a9f0d332953bcf9e7184688985211d041e4fdb8c09535cfbb3: Status 404 returned error can't find the container with id f20153659601b8a9f0d332953bcf9e7184688985211d041e4fdb8c09535cfbb3 Oct 30 12:14:15.089852 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:15.089768 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f409f1-8fa9-438b-b05e-128b59362c24" path="/var/lib/kubelet/pods/28f409f1-8fa9-438b-b05e-128b59362c24/volumes" Oct 30 12:14:15.090176 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:15.090163 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60230ddf-11ad-4c9d-b49e-9d2547dc929e" path="/var/lib/kubelet/pods/60230ddf-11ad-4c9d-b49e-9d2547dc929e/volumes" Oct 30 12:14:15.090559 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:15.090547 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5e74d16-3dde-444b-8f10-769314d02026" path="/var/lib/kubelet/pods/d5e74d16-3dde-444b-8f10-769314d02026/volumes" Oct 30 12:14:15.572812 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:15.572774 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" event={"ID":"4572318d-2ff5-43d0-9071-faeb6e94c178","Type":"ContainerStarted","Data":"707155a85afee81f8881e08418e961fccbb2123b7f25ff1b8d0e34d273172f56"} Oct 30 12:14:15.573357 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:15.572833 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" event={"ID":"4572318d-2ff5-43d0-9071-faeb6e94c178","Type":"ContainerStarted","Data":"87ff20470cb3380054592a2b8b9a4e81e7b7acdb161c3b4bc1e016f132c7175b"} Oct 30 12:14:15.573357 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:15.572954 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" Oct 30 12:14:15.574572 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:15.574545 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" event={"ID":"127e9882-c84b-4162-9565-834a8126f9e3","Type":"ContainerStarted","Data":"645e122f975e1e42c9c9766f80a4b290047cb9b7e75429a172953a51a91663a8"} Oct 30 12:14:15.574698 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:15.574594 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" event={"ID":"127e9882-c84b-4162-9565-834a8126f9e3","Type":"ContainerStarted","Data":"f20153659601b8a9f0d332953bcf9e7184688985211d041e4fdb8c09535cfbb3"} Oct 30 12:14:15.574698 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:15.574672 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" Oct 30 12:14:15.613503 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:15.613448 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" podStartSLOduration=1.613432307 podStartE2EDuration="1.613432307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:15.612402121 +0000 UTC m=+1206.646540519" watchObservedRunningTime="2026-10-30 12:14:15.613432307 +0000 UTC m=+1206.647570697" Oct 30 12:14:15.653204 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:15.653145 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" podStartSLOduration=1.6531269100000001 podStartE2EDuration="1.65312691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:15.651964167 +0000 UTC m=+1206.686102563" watchObservedRunningTime="2026-10-30 12:14:15.65312691 +0000 UTC m=+1206.687265305" Oct 30 12:14:15.858907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:15.858810 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4"] Oct 30 12:14:15.859157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:15.859136 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" podUID="2b142407-8432-4ed2-bb39-2744274f83b5" containerName="maas-api" containerID="cri-o://f76162d27fb7d081ef59b5c96a6fc57848dd7eae2f3431ba7bb8f7bdb8a92e55" gracePeriod=30 Oct 30 12:14:16.167158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.167120 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" Oct 30 12:14:16.209123 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.209090 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b142407-8432-4ed2-bb39-2744274f83b5-kube-api-access-lt9cw\" (UniqueName: \"kubernetes.io/projected/2b142407-8432-4ed2-bb39-2744274f83b5-kube-api-access-lt9cw\") pod \"2b142407-8432-4ed2-bb39-2744274f83b5\" (UID: \"2b142407-8432-4ed2-bb39-2744274f83b5\") " Oct 30 12:14:16.209324 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.209182 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b142407-8432-4ed2-bb39-2744274f83b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b142407-8432-4ed2-bb39-2744274f83b5-openshift-service-ca\") pod \"2b142407-8432-4ed2-bb39-2744274f83b5\" (UID: \"2b142407-8432-4ed2-bb39-2744274f83b5\") " Oct 30 12:14:16.209324 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.209266 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b142407-8432-4ed2-bb39-2744274f83b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b142407-8432-4ed2-bb39-2744274f83b5-maas-api-tls\") pod \"2b142407-8432-4ed2-bb39-2744274f83b5\" (UID: \"2b142407-8432-4ed2-bb39-2744274f83b5\") " Oct 30 12:14:16.209440 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.209372 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b142407-8432-4ed2-bb39-2744274f83b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b142407-8432-4ed2-bb39-2744274f83b5-metrics-certs\") pod \"2b142407-8432-4ed2-bb39-2744274f83b5\" (UID: \"2b142407-8432-4ed2-bb39-2744274f83b5\") " Oct 30 12:14:16.209888 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.209856 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b142407-8432-4ed2-bb39-2744274f83b5-openshift-service-ca" pod "2b142407-8432-4ed2-bb39-2744274f83b5" (UID: "2b142407-8432-4ed2-bb39-2744274f83b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:14:16.211708 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.211677 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b142407-8432-4ed2-bb39-2744274f83b5-kube-api-access-lt9cw" pod "2b142407-8432-4ed2-bb39-2744274f83b5" (UID: "2b142407-8432-4ed2-bb39-2744274f83b5"). InnerVolumeSpecName "kube-api-access-lt9cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:16.212246 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.212212 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b142407-8432-4ed2-bb39-2744274f83b5-maas-api-tls" pod "2b142407-8432-4ed2-bb39-2744274f83b5" (UID: "2b142407-8432-4ed2-bb39-2744274f83b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:14:16.212437 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.212412 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b142407-8432-4ed2-bb39-2744274f83b5-metrics-certs" pod "2b142407-8432-4ed2-bb39-2744274f83b5" (UID: "2b142407-8432-4ed2-bb39-2744274f83b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:14:16.310321 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.310289 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b142407-8432-4ed2-bb39-2744274f83b5-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:16.310321 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.310319 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt9cw\" (UniqueName: \"kubernetes.io/projected/2b142407-8432-4ed2-bb39-2744274f83b5-kube-api-access-lt9cw\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:16.310551 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.310330 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b142407-8432-4ed2-bb39-2744274f83b5-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:16.310551 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.310340 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b142407-8432-4ed2-bb39-2744274f83b5-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:16.576342 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.576311 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4"] Oct 30 12:14:16.576961 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.576941 4468 cpu_manager.go:414] "containerMap: removing container" podUID="2b142407-8432-4ed2-bb39-2744274f83b5" containerName="maas-api" Oct 30 12:14:16.577045 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.576964 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b142407-8432-4ed2-bb39-2744274f83b5" containerName="maas-api" Oct 30 12:14:16.577141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.577099 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b142407-8432-4ed2-bb39-2744274f83b5" containerName="maas-api" Oct 30 12:14:16.579248 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.579221 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" Oct 30 12:14:16.581886 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.581859 4468 generic.go:356] "Generic (PLEG): container finished" podID="2b142407-8432-4ed2-bb39-2744274f83b5" containerID="f76162d27fb7d081ef59b5c96a6fc57848dd7eae2f3431ba7bb8f7bdb8a92e55" exitCode=0 Oct 30 12:14:16.581997 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.581915 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" Oct 30 12:14:16.581997 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.581951 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" event={"ID":"2b142407-8432-4ed2-bb39-2744274f83b5","Type":"ContainerDied","Data":"f76162d27fb7d081ef59b5c96a6fc57848dd7eae2f3431ba7bb8f7bdb8a92e55"} Oct 30 12:14:16.582127 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.582014 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4" event={"ID":"2b142407-8432-4ed2-bb39-2744274f83b5","Type":"ContainerDied","Data":"e45efaa945c5ba9b2ad4fbab0075cfac2f9af7c5153d551195ddcd5115efec6d"} Oct 30 12:14:16.582127 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.582042 4468 scope.go:122] "RemoveContainer" containerID="f76162d27fb7d081ef59b5c96a6fc57848dd7eae2f3431ba7bb8f7bdb8a92e55" Oct 30 12:14:16.593774 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.593744 4468 scope.go:122] "RemoveContainer" containerID="f76162d27fb7d081ef59b5c96a6fc57848dd7eae2f3431ba7bb8f7bdb8a92e55" Oct 30 12:14:16.595404 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:14:16.595368 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76162d27fb7d081ef59b5c96a6fc57848dd7eae2f3431ba7bb8f7bdb8a92e55\": container with ID starting with f76162d27fb7d081ef59b5c96a6fc57848dd7eae2f3431ba7bb8f7bdb8a92e55 not found: ID does not exist" containerID="f76162d27fb7d081ef59b5c96a6fc57848dd7eae2f3431ba7bb8f7bdb8a92e55" Oct 30 12:14:16.595549 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.595421 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76162d27fb7d081ef59b5c96a6fc57848dd7eae2f3431ba7bb8f7bdb8a92e55"} err="failed to get container status \"f76162d27fb7d081ef59b5c96a6fc57848dd7eae2f3431ba7bb8f7bdb8a92e55\": rpc error: code = NotFound desc = could not find container \"f76162d27fb7d081ef59b5c96a6fc57848dd7eae2f3431ba7bb8f7bdb8a92e55\": container with ID starting with f76162d27fb7d081ef59b5c96a6fc57848dd7eae2f3431ba7bb8f7bdb8a92e55 not found: ID does not exist" Oct 30 12:14:16.602138 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.599467 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f920f7\"" Oct 30 12:14:16.604057 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.604029 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4"] Oct 30 12:14:16.642969 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.642871 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2"] Oct 30 12:14:16.645782 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.645758 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" Oct 30 12:14:16.669389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.669354 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2"] Oct 30 12:14:16.673057 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.673024 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4"] Oct 30 12:14:16.676192 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.676157 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-dfd96f0e-6f4599565-fg7j4"] Oct 30 12:14:16.707103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.707055 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f920f7-dockercfg-k4dgv\"" Oct 30 12:14:16.716767 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.716722 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29813ba6-6304-4a2f-ab9b-ce9f4d904ad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2\" (UID: \"29813ba6-6304-4a2f-ab9b-ce9f4d904ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" Oct 30 12:14:16.716950 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.716927 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq79k\" (UniqueName: \"kubernetes.io/projected/29813ba6-6304-4a2f-ab9b-ce9f4d904ad5-kube-api-access-tq79k\") pod \"payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2\" (UID: \"29813ba6-6304-4a2f-ab9b-ce9f4d904ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" Oct 30 12:14:16.717035 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.716967 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq54q\" (UniqueName: \"kubernetes.io/projected/484b8ef9-46da-48da-a050-97bc7cea4f07-kube-api-access-vq54q\") pod \"payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4\" (UID: \"484b8ef9-46da-48da-a050-97bc7cea4f07\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" Oct 30 12:14:16.717035 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.717023 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/484b8ef9-46da-48da-a050-97bc7cea4f07-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4\" (UID: \"484b8ef9-46da-48da-a050-97bc7cea4f07\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" Oct 30 12:14:16.818828 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.818787 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29813ba6-6304-4a2f-ab9b-ce9f4d904ad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2\" (UID: \"29813ba6-6304-4a2f-ab9b-ce9f4d904ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" Oct 30 12:14:16.819012 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.818883 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq79k\" (UniqueName: \"kubernetes.io/projected/29813ba6-6304-4a2f-ab9b-ce9f4d904ad5-kube-api-access-tq79k\") pod \"payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2\" (UID: \"29813ba6-6304-4a2f-ab9b-ce9f4d904ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" Oct 30 12:14:16.819012 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.818916 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq54q\" (UniqueName: \"kubernetes.io/projected/484b8ef9-46da-48da-a050-97bc7cea4f07-kube-api-access-vq54q\") pod \"payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4\" (UID: \"484b8ef9-46da-48da-a050-97bc7cea4f07\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" Oct 30 12:14:16.819169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.819044 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/484b8ef9-46da-48da-a050-97bc7cea4f07-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4\" (UID: \"484b8ef9-46da-48da-a050-97bc7cea4f07\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" Oct 30 12:14:16.819651 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.819627 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29813ba6-6304-4a2f-ab9b-ce9f4d904ad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2\" (UID: \"29813ba6-6304-4a2f-ab9b-ce9f4d904ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" Oct 30 12:14:16.819803 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.819783 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/484b8ef9-46da-48da-a050-97bc7cea4f07-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4\" (UID: \"484b8ef9-46da-48da-a050-97bc7cea4f07\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" Oct 30 12:14:16.844734 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.844645 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq54q\" (UniqueName: \"kubernetes.io/projected/484b8ef9-46da-48da-a050-97bc7cea4f07-kube-api-access-vq54q\") pod \"payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4\" (UID: \"484b8ef9-46da-48da-a050-97bc7cea4f07\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" Oct 30 12:14:16.845319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.845289 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq79k\" (UniqueName: \"kubernetes.io/projected/29813ba6-6304-4a2f-ab9b-ce9f4d904ad5-kube-api-access-tq79k\") pod \"payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2\" (UID: \"29813ba6-6304-4a2f-ab9b-ce9f4d904ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" Oct 30 12:14:16.861096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.861047 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h"] Oct 30 12:14:16.861329 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.861302 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" podUID="f1338306-3768-445a-a6eb-57eafbbeeed0" containerName="istio-proxy" containerID="cri-o://3be1247efe812dadc1bcdc144f3b5e4a0f7f67bfeb6faf42f5ba7c92cdff28c2" gracePeriod=30 Oct 30 12:14:16.903810 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.903774 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" Oct 30 12:14:16.959034 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:16.958965 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" Oct 30 12:14:17.096237 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:17.096117 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b142407-8432-4ed2-bb39-2744274f83b5" path="/var/lib/kubelet/pods/2b142407-8432-4ed2-bb39-2744274f83b5/volumes" Oct 30 12:14:17.214025 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:17.213995 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4"] Oct 30 12:14:17.214249 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:14:17.214225 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484b8ef9_46da_48da_a050_97bc7cea4f07.slice/crio-3c6ceb6f497ba022fa3a6587103d9579eec3388a75c3d2dce9fc1b3220d04ac7 WatchSource:0}: Error finding container 3c6ceb6f497ba022fa3a6587103d9579eec3388a75c3d2dce9fc1b3220d04ac7: Status 404 returned error can't find the container with id 3c6ceb6f497ba022fa3a6587103d9579eec3388a75c3d2dce9fc1b3220d04ac7 Oct 30 12:14:17.337931 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:17.337892 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2"] Oct 30 12:14:17.371090 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:14:17.369397 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29813ba6_6304_4a2f_ab9b_ce9f4d904ad5.slice/crio-30ee0e57206074e3934e0a17a95c00fa2048c27676c3d53cfff2b93cd3346db0 WatchSource:0}: Error finding container 30ee0e57206074e3934e0a17a95c00fa2048c27676c3d53cfff2b93cd3346db0: Status 404 returned error can't find the container with id 30ee0e57206074e3934e0a17a95c00fa2048c27676c3d53cfff2b93cd3346db0 Oct 30 12:14:17.595537 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:17.595499 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" event={"ID":"29813ba6-6304-4a2f-ab9b-ce9f4d904ad5","Type":"ContainerStarted","Data":"1c5bbc7ad4934a71a5d0ca3f51a3f3bae5fc91c74d80f1468eaab744e706eb8b"} Oct 30 12:14:17.596057 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:17.595562 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" event={"ID":"29813ba6-6304-4a2f-ab9b-ce9f4d904ad5","Type":"ContainerStarted","Data":"30ee0e57206074e3934e0a17a95c00fa2048c27676c3d53cfff2b93cd3346db0"} Oct 30 12:14:17.596057 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:17.595744 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" Oct 30 12:14:17.598712 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:17.598666 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" event={"ID":"484b8ef9-46da-48da-a050-97bc7cea4f07","Type":"ContainerStarted","Data":"549e66280e29ec55f73a3cf8acf4704ec48170c7bc9b42a98460d7d2e2074d2b"} Oct 30 12:14:17.598859 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:17.598736 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" event={"ID":"484b8ef9-46da-48da-a050-97bc7cea4f07","Type":"ContainerStarted","Data":"3c6ceb6f497ba022fa3a6587103d9579eec3388a75c3d2dce9fc1b3220d04ac7"} Oct 30 12:14:17.598859 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:17.598808 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" Oct 30 12:14:17.635966 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:17.635853 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" podStartSLOduration=1.6358380829999999 podStartE2EDuration="1.635838083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:17.634578714 +0000 UTC m=+1208.668717110" watchObservedRunningTime="2026-10-30 12:14:17.635838083 +0000 UTC m=+1208.669976479" Oct 30 12:14:17.678708 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:17.678650 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" podStartSLOduration=1.678632675 podStartE2EDuration="1.678632675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:17.6760993 +0000 UTC m=+1208.710237689" watchObservedRunningTime="2026-10-30 12:14:17.678632675 +0000 UTC m=+1208.712771121" Oct 30 12:14:18.197567 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.197533 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl"] Oct 30 12:14:18.200274 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.200247 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.219890 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.219857 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl"] Oct 30 12:14:18.229110 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.227673 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c1b1e8-openshift-default-dockercfg-k4lgw\"" Oct 30 12:14:18.236418 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.236384 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.236418 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.236421 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.236696 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.236510 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.236696 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.236589 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx528\" (UniqueName: \"kubernetes.io/projected/56010c34-d070-4709-8ec7-6c9a23178226-kube-api-access-jx528\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.236696 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.236634 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56010c34-d070-4709-8ec7-6c9a23178226-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.236696 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.236657 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.236696 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.236695 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56010c34-d070-4709-8ec7-6c9a23178226-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.236965 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.236754 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56010c34-d070-4709-8ec7-6c9a23178226-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.236965 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.236852 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.337810 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.337767 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx528\" (UniqueName: \"kubernetes.io/projected/56010c34-d070-4709-8ec7-6c9a23178226-kube-api-access-jx528\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.337967 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.337831 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56010c34-d070-4709-8ec7-6c9a23178226-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.337967 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.337865 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.337967 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.337894 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56010c34-d070-4709-8ec7-6c9a23178226-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.337967 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.337917 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56010c34-d070-4709-8ec7-6c9a23178226-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.337967 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.337947 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.338213 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.338022 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.338213 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.338108 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.338213 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.338199 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.338387 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.338352 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.338458 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.338415 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.338580 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.338541 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.338691 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.338559 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.338954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.338933 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56010c34-d070-4709-8ec7-6c9a23178226-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.340889 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.340856 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.341012 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.340959 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56010c34-d070-4709-8ec7-6c9a23178226-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.395105 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.394368 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56010c34-d070-4709-8ec7-6c9a23178226-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.400890 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.400812 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx528\" (UniqueName: \"kubernetes.io/projected/56010c34-d070-4709-8ec7-6c9a23178226-kube-api-access-jx528\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.516448 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.516365 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:18.855848 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.855817 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl"] Oct 30 12:14:18.857085 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:14:18.857042 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56010c34_d070_4709_8ec7_6c9a23178226.slice/crio-cd7bc9cfcdf2698cce940a334414165e069a32b6d85aa3b86c50758e7f8f0420 WatchSource:0}: Error finding container cd7bc9cfcdf2698cce940a334414165e069a32b6d85aa3b86c50758e7f8f0420: Status 404 returned error can't find the container with id cd7bc9cfcdf2698cce940a334414165e069a32b6d85aa3b86c50758e7f8f0420 Oct 30 12:14:18.859656 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.859623 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:14:18.859761 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.859712 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:14:18.859818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:18.859784 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:14:19.507400 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:19.507368 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" Oct 30 12:14:19.621816 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:19.621771 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" event={"ID":"56010c34-d070-4709-8ec7-6c9a23178226","Type":"ContainerStarted","Data":"5eda8ae7d53781a7002538e18113ad4943607ce95403afeab63c01b68a12ba20"} Oct 30 12:14:19.622039 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:19.621877 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" event={"ID":"56010c34-d070-4709-8ec7-6c9a23178226","Type":"ContainerStarted","Data":"cd7bc9cfcdf2698cce940a334414165e069a32b6d85aa3b86c50758e7f8f0420"} Oct 30 12:14:19.670471 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:19.670396 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" podStartSLOduration=1.670372518 podStartE2EDuration="1.670372518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:19.666449257 +0000 UTC m=+1210.700587674" watchObservedRunningTime="2026-10-30 12:14:19.670372518 +0000 UTC m=+1210.704510914" Oct 30 12:14:20.517357 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:20.517321 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:20.523566 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:20.523532 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:20.627729 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:20.627690 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:20.628836 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:20.628814 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:21.914534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:21.914486 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl"] Oct 30 12:14:21.917802 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:21.917774 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" Oct 30 12:14:21.944282 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:21.944246 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-f920f7-serving-cert\"" Oct 30 12:14:21.944502 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:21.944280 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 12:14:21.944502 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:21.944375 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rtvpm\"" Oct 30 12:14:21.949714 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:21.949618 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl"] Oct 30 12:14:22.022229 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.022106 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl\" (UID: \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" Oct 30 12:14:22.022229 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.022167 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl\" (UID: \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" Oct 30 12:14:22.022229 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.022199 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnpdh\" (UniqueName: \"kubernetes.io/projected/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-kube-api-access-hnpdh\") pod \"maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl\" (UID: \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" Oct 30 12:14:22.022229 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.022236 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-metrics-certs\") pod \"maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl\" (UID: \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" Oct 30 12:14:22.142311 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.142265 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-metrics-certs\") pod \"maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl\" (UID: \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" Oct 30 12:14:22.142509 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.142452 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl\" (UID: \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" Oct 30 12:14:22.142509 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.142491 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl\" (UID: \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" Oct 30 12:14:22.142634 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.142547 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnpdh\" (UniqueName: \"kubernetes.io/projected/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-kube-api-access-hnpdh\") pod \"maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl\" (UID: \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" Oct 30 12:14:22.145222 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:14:22.145179 4468 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-serving-cert: secret "maas-api-e2e-worker-w0-f920f7-serving-cert" not found Oct 30 12:14:22.145369 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:14:22.145263 4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-maas-api-tls podName:b7a743c1-ed5e-4fec-aa03-e3fb9f10f048 nodeName:}" failed. No retries permitted until 2026-10-30 12:14:22.645243497 +0000 UTC m=+1213.679381886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-maas-api-tls") pod "maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" (UID: "b7a743c1-ed5e-4fec-aa03-e3fb9f10f048") : secret "maas-api-e2e-worker-w0-f920f7-serving-cert" not found Oct 30 12:14:22.145369 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.145326 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl\" (UID: \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" Oct 30 12:14:22.156176 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.149654 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-metrics-certs\") pod \"maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl\" (UID: \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" Oct 30 12:14:22.200542 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.200475 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnpdh\" (UniqueName: \"kubernetes.io/projected/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-kube-api-access-hnpdh\") pod \"maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl\" (UID: \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" Oct 30 12:14:22.233461 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.233430 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:14:22.344697 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.344659 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1338306-3768-445a-a6eb-57eafbbeeed0-kube-api-access-dgwfp\" (UniqueName: \"kubernetes.io/projected/f1338306-3768-445a-a6eb-57eafbbeeed0-kube-api-access-dgwfp\") pod \"f1338306-3768-445a-a6eb-57eafbbeeed0\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " Oct 30 12:14:22.344900 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.344763 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-workload-socket\") pod \"f1338306-3768-445a-a6eb-57eafbbeeed0\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " Oct 30 12:14:22.344900 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.344807 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-data\") pod \"f1338306-3768-445a-a6eb-57eafbbeeed0\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " Oct 30 12:14:22.345030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.344924 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-token\" (UniqueName: \"kubernetes.io/projected/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-token\") pod \"f1338306-3768-445a-a6eb-57eafbbeeed0\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " Oct 30 12:14:22.345030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.344969 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-podinfo\") pod \"f1338306-3768-445a-a6eb-57eafbbeeed0\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " Oct 30 12:14:22.345181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.345031 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1338306-3768-445a-a6eb-57eafbbeeed0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1338306-3768-445a-a6eb-57eafbbeeed0-istiod-ca-cert\") pod \"f1338306-3768-445a-a6eb-57eafbbeeed0\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " Oct 30 12:14:22.345181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.345088 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-credential-socket\") pod \"f1338306-3768-445a-a6eb-57eafbbeeed0\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " Oct 30 12:14:22.345181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.345152 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-workload-certs\") pod \"f1338306-3768-445a-a6eb-57eafbbeeed0\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " Oct 30 12:14:22.345326 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.345169 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-workload-socket" pod "f1338306-3768-445a-a6eb-57eafbbeeed0" (UID: "f1338306-3768-445a-a6eb-57eafbbeeed0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:22.345326 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.345201 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-envoy\") pod \"f1338306-3768-445a-a6eb-57eafbbeeed0\" (UID: \"f1338306-3768-445a-a6eb-57eafbbeeed0\") " Oct 30 12:14:22.345326 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.345277 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-data" pod "f1338306-3768-445a-a6eb-57eafbbeeed0" (UID: "f1338306-3768-445a-a6eb-57eafbbeeed0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:22.345455 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.345389 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-credential-socket" pod "f1338306-3768-445a-a6eb-57eafbbeeed0" (UID: "f1338306-3768-445a-a6eb-57eafbbeeed0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:22.345568 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.345538 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-workload-certs" pod "f1338306-3768-445a-a6eb-57eafbbeeed0" (UID: "f1338306-3768-445a-a6eb-57eafbbeeed0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:22.345711 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.345643 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1338306-3768-445a-a6eb-57eafbbeeed0-istiod-ca-cert" pod "f1338306-3768-445a-a6eb-57eafbbeeed0" (UID: "f1338306-3768-445a-a6eb-57eafbbeeed0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:14:22.345773 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.345743 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1338306-3768-445a-a6eb-57eafbbeeed0-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:22.345773 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.345765 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:22.345914 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.345780 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:22.345914 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.345794 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:22.345914 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.345806 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:22.347629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.347603 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-token" pod "f1338306-3768-445a-a6eb-57eafbbeeed0" (UID: "f1338306-3768-445a-a6eb-57eafbbeeed0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:22.347731 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.347644 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1338306-3768-445a-a6eb-57eafbbeeed0-kube-api-access-dgwfp" pod "f1338306-3768-445a-a6eb-57eafbbeeed0" (UID: "f1338306-3768-445a-a6eb-57eafbbeeed0"). InnerVolumeSpecName "kube-api-access-dgwfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:22.347731 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.347646 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-podinfo" pod "f1338306-3768-445a-a6eb-57eafbbeeed0" (UID: "f1338306-3768-445a-a6eb-57eafbbeeed0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:14:22.347859 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.347837 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-envoy" pod "f1338306-3768-445a-a6eb-57eafbbeeed0" (UID: "f1338306-3768-445a-a6eb-57eafbbeeed0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:22.446897 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.446852 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:22.446897 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.446894 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:22.447176 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.446909 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1338306-3768-445a-a6eb-57eafbbeeed0-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:22.447176 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.446922 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgwfp\" (UniqueName: \"kubernetes.io/projected/f1338306-3768-445a-a6eb-57eafbbeeed0-kube-api-access-dgwfp\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:14:22.641629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.641591 4468 generic.go:356] "Generic (PLEG): container finished" podID="f1338306-3768-445a-a6eb-57eafbbeeed0" containerID="3be1247efe812dadc1bcdc144f3b5e4a0f7f67bfeb6faf42f5ba7c92cdff28c2" exitCode=0 Oct 30 12:14:22.641843 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.641659 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" Oct 30 12:14:22.641843 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.641667 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" event={"ID":"f1338306-3768-445a-a6eb-57eafbbeeed0","Type":"ContainerDied","Data":"3be1247efe812dadc1bcdc144f3b5e4a0f7f67bfeb6faf42f5ba7c92cdff28c2"} Oct 30 12:14:22.641843 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.641737 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h" event={"ID":"f1338306-3768-445a-a6eb-57eafbbeeed0","Type":"ContainerDied","Data":"0d59d2976c5e113eeb1db1aa17c2fbb34565c7d0b73c71fefc5aa440cd4b0845"} Oct 30 12:14:22.641843 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.641767 4468 scope.go:122] "RemoveContainer" containerID="3be1247efe812dadc1bcdc144f3b5e4a0f7f67bfeb6faf42f5ba7c92cdff28c2" Oct 30 12:14:22.649134 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.649103 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl\" (UID: \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" Oct 30 12:14:22.652012 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.651918 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl\" (UID: \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" Oct 30 12:14:22.652160 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.652142 4468 scope.go:122] "RemoveContainer" containerID="a3e2fd880efa5ac2f6c9d8bf7cd05fc9920a720141df55c85cf00a034d3a339e" Oct 30 12:14:22.662269 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.662242 4468 scope.go:122] "RemoveContainer" containerID="3be1247efe812dadc1bcdc144f3b5e4a0f7f67bfeb6faf42f5ba7c92cdff28c2" Oct 30 12:14:22.662627 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:14:22.662603 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3be1247efe812dadc1bcdc144f3b5e4a0f7f67bfeb6faf42f5ba7c92cdff28c2\": container with ID starting with 3be1247efe812dadc1bcdc144f3b5e4a0f7f67bfeb6faf42f5ba7c92cdff28c2 not found: ID does not exist" containerID="3be1247efe812dadc1bcdc144f3b5e4a0f7f67bfeb6faf42f5ba7c92cdff28c2" Oct 30 12:14:22.662784 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.662635 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be1247efe812dadc1bcdc144f3b5e4a0f7f67bfeb6faf42f5ba7c92cdff28c2"} err="failed to get container status \"3be1247efe812dadc1bcdc144f3b5e4a0f7f67bfeb6faf42f5ba7c92cdff28c2\": rpc error: code = NotFound desc = could not find container \"3be1247efe812dadc1bcdc144f3b5e4a0f7f67bfeb6faf42f5ba7c92cdff28c2\": container with ID starting with 3be1247efe812dadc1bcdc144f3b5e4a0f7f67bfeb6faf42f5ba7c92cdff28c2 not found: ID does not exist" Oct 30 12:14:22.662784 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.662653 4468 scope.go:122] "RemoveContainer" containerID="a3e2fd880efa5ac2f6c9d8bf7cd05fc9920a720141df55c85cf00a034d3a339e" Oct 30 12:14:22.662931 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:14:22.662911 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e2fd880efa5ac2f6c9d8bf7cd05fc9920a720141df55c85cf00a034d3a339e\": container with ID starting with a3e2fd880efa5ac2f6c9d8bf7cd05fc9920a720141df55c85cf00a034d3a339e not found: ID does not exist" containerID="a3e2fd880efa5ac2f6c9d8bf7cd05fc9920a720141df55c85cf00a034d3a339e" Oct 30 12:14:22.662984 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.662937 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e2fd880efa5ac2f6c9d8bf7cd05fc9920a720141df55c85cf00a034d3a339e"} err="failed to get container status \"a3e2fd880efa5ac2f6c9d8bf7cd05fc9920a720141df55c85cf00a034d3a339e\": rpc error: code = NotFound desc = could not find container \"a3e2fd880efa5ac2f6c9d8bf7cd05fc9920a720141df55c85cf00a034d3a339e\": container with ID starting with a3e2fd880efa5ac2f6c9d8bf7cd05fc9920a720141df55c85cf00a034d3a339e not found: ID does not exist" Oct 30 12:14:22.701036 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.700931 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h"] Oct 30 12:14:22.712687 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.712647 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-dfd96f0e-openshift-default-6d865cbfdf-mjw6h"] Oct 30 12:14:22.873439 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:22.873404 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" Oct 30 12:14:23.089274 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:23.089192 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1338306-3768-445a-a6eb-57eafbbeeed0" path="/var/lib/kubelet/pods/f1338306-3768-445a-a6eb-57eafbbeeed0/volumes" Oct 30 12:14:23.181779 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:23.181657 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl"] Oct 30 12:14:23.181779 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:14:23.181682 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a743c1_ed5e_4fec_aa03_e3fb9f10f048.slice/crio-62c4e48519a3ae04bdad9f5d8cba47042176593168f696973b796d91954aa835 WatchSource:0}: Error finding container 62c4e48519a3ae04bdad9f5d8cba47042176593168f696973b796d91954aa835: Status 404 returned error can't find the container with id 62c4e48519a3ae04bdad9f5d8cba47042176593168f696973b796d91954aa835 Oct 30 12:14:23.558465 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:23.558434 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" Oct 30 12:14:23.600891 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:14:23.600837 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:14:23.650621 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:23.650581 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" event={"ID":"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048","Type":"ContainerStarted","Data":"c5ba133dc68ab2963cc8757ca0f036f22700f307c6e89813169b310defea564c"} Oct 30 12:14:23.650800 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:23.650643 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" event={"ID":"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048","Type":"ContainerStarted","Data":"62c4e48519a3ae04bdad9f5d8cba47042176593168f696973b796d91954aa835"} Oct 30 12:14:23.650800 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:23.650674 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" Oct 30 12:14:23.691580 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:23.691527 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" podStartSLOduration=2.40938296 podStartE2EDuration="2.69151224s" totalImagesPullingTime="282.12928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:23.690825947 +0000 UTC m=+1214.724964343" watchObservedRunningTime="2026-10-30 12:14:23.69151224 +0000 UTC m=+1214.725650689" Oct 30 12:14:24.077513 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.077470 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7"] Oct 30 12:14:24.077995 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.077977 4468 cpu_manager.go:414] "containerMap: removing container" podUID="f1338306-3768-445a-a6eb-57eafbbeeed0" containerName="istio-proxy" Oct 30 12:14:24.078123 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.077998 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1338306-3768-445a-a6eb-57eafbbeeed0" containerName="istio-proxy" Oct 30 12:14:24.078231 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.078216 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1338306-3768-445a-a6eb-57eafbbeeed0" containerName="istio-proxy" Oct 30 12:14:24.078293 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.078236 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1338306-3768-445a-a6eb-57eafbbeeed0" containerName="istio-proxy" Oct 30 12:14:24.080532 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.080509 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" Oct 30 12:14:24.115225 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.115187 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-c1b1e8-dockercfg-zxfvx\"" Oct 30 12:14:24.115915 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.115896 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-c1b1e8\"" Oct 30 12:14:24.159095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.158896 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7"] Oct 30 12:14:24.191038 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.190997 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qfpt\" (UniqueName: \"kubernetes.io/projected/2dbcae46-3576-48d9-9691-50ddfaefd50e-kube-api-access-5qfpt\") pod \"payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7\" (UID: \"2dbcae46-3576-48d9-9691-50ddfaefd50e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" Oct 30 12:14:24.196892 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.196840 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbcae46-3576-48d9-9691-50ddfaefd50e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7\" (UID: \"2dbcae46-3576-48d9-9691-50ddfaefd50e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" Oct 30 12:14:24.200877 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.200831 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp"] Oct 30 12:14:24.201387 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.201371 4468 cpu_manager.go:414] "containerMap: removing container" podUID="f1338306-3768-445a-a6eb-57eafbbeeed0" containerName="istio-proxy" Oct 30 12:14:24.201473 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.201389 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1338306-3768-445a-a6eb-57eafbbeeed0" containerName="istio-proxy" Oct 30 12:14:24.203688 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.203659 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" Oct 30 12:14:24.217287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.217253 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp"] Oct 30 12:14:24.298723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.298687 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qfpt\" (UniqueName: \"kubernetes.io/projected/2dbcae46-3576-48d9-9691-50ddfaefd50e-kube-api-access-5qfpt\") pod \"payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7\" (UID: \"2dbcae46-3576-48d9-9691-50ddfaefd50e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" Oct 30 12:14:24.298924 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.298739 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4p42\" (UniqueName: \"kubernetes.io/projected/42777387-98e9-4718-aa0e-dcdcb64b7a19-kube-api-access-m4p42\") pod \"payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp\" (UID: \"42777387-98e9-4718-aa0e-dcdcb64b7a19\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" Oct 30 12:14:24.298924 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.298816 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42777387-98e9-4718-aa0e-dcdcb64b7a19-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp\" (UID: \"42777387-98e9-4718-aa0e-dcdcb64b7a19\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" Oct 30 12:14:24.299040 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.298929 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbcae46-3576-48d9-9691-50ddfaefd50e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7\" (UID: \"2dbcae46-3576-48d9-9691-50ddfaefd50e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" Oct 30 12:14:24.299703 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.299684 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbcae46-3576-48d9-9691-50ddfaefd50e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7\" (UID: \"2dbcae46-3576-48d9-9691-50ddfaefd50e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" Oct 30 12:14:24.321788 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.321748 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qfpt\" (UniqueName: \"kubernetes.io/projected/2dbcae46-3576-48d9-9691-50ddfaefd50e-kube-api-access-5qfpt\") pod \"payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7\" (UID: \"2dbcae46-3576-48d9-9691-50ddfaefd50e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" Oct 30 12:14:24.392511 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.392422 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" Oct 30 12:14:24.400240 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.400190 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42777387-98e9-4718-aa0e-dcdcb64b7a19-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp\" (UID: \"42777387-98e9-4718-aa0e-dcdcb64b7a19\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" Oct 30 12:14:24.400403 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.400386 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4p42\" (UniqueName: \"kubernetes.io/projected/42777387-98e9-4718-aa0e-dcdcb64b7a19-kube-api-access-m4p42\") pod \"payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp\" (UID: \"42777387-98e9-4718-aa0e-dcdcb64b7a19\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" Oct 30 12:14:24.400996 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.400974 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42777387-98e9-4718-aa0e-dcdcb64b7a19-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp\" (UID: \"42777387-98e9-4718-aa0e-dcdcb64b7a19\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" Oct 30 12:14:24.423554 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.423524 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4p42\" (UniqueName: \"kubernetes.io/projected/42777387-98e9-4718-aa0e-dcdcb64b7a19-kube-api-access-m4p42\") pod \"payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp\" (UID: \"42777387-98e9-4718-aa0e-dcdcb64b7a19\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" Oct 30 12:14:24.520871 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.520830 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" Oct 30 12:14:24.569409 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.569372 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" Oct 30 12:14:24.695921 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.695888 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7"] Oct 30 12:14:24.697209 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:14:24.697177 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbcae46_3576_48d9_9691_50ddfaefd50e.slice/crio-8efb43462b922332ca1a4124775fa42103b5ed32c40d8144511a4d9ac0a4bcd0 WatchSource:0}: Error finding container 8efb43462b922332ca1a4124775fa42103b5ed32c40d8144511a4d9ac0a4bcd0: Status 404 returned error can't find the container with id 8efb43462b922332ca1a4124775fa42103b5ed32c40d8144511a4d9ac0a4bcd0 Oct 30 12:14:24.824204 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:24.824168 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp"] Oct 30 12:14:24.825523 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:14:24.825497 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42777387_98e9_4718_aa0e_dcdcb64b7a19.slice/crio-737e86857844149f7f9ab458fd8ac3bf4e6bc43df7f3a8dc38a371d92f4d2e4b WatchSource:0}: Error finding container 737e86857844149f7f9ab458fd8ac3bf4e6bc43df7f3a8dc38a371d92f4d2e4b: Status 404 returned error can't find the container with id 737e86857844149f7f9ab458fd8ac3bf4e6bc43df7f3a8dc38a371d92f4d2e4b Oct 30 12:14:25.665061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:25.665017 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" event={"ID":"2dbcae46-3576-48d9-9691-50ddfaefd50e","Type":"ContainerStarted","Data":"23e5ef21dca0e9e75d2ce2fe9ebd7d8406c3163ad3d33f32eab5d5904f7c04f0"} Oct 30 12:14:25.665592 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:25.665092 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" event={"ID":"2dbcae46-3576-48d9-9691-50ddfaefd50e","Type":"ContainerStarted","Data":"8efb43462b922332ca1a4124775fa42103b5ed32c40d8144511a4d9ac0a4bcd0"} Oct 30 12:14:25.665592 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:25.665115 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" Oct 30 12:14:25.666685 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:25.666657 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" event={"ID":"42777387-98e9-4718-aa0e-dcdcb64b7a19","Type":"ContainerStarted","Data":"af1a9bc7dbe714cc2d93960e8d320b235695947cda49bae0e6b48c038d4b6401"} Oct 30 12:14:25.666797 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:25.666715 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" Oct 30 12:14:25.666797 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:25.666738 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" event={"ID":"42777387-98e9-4718-aa0e-dcdcb64b7a19","Type":"ContainerStarted","Data":"737e86857844149f7f9ab458fd8ac3bf4e6bc43df7f3a8dc38a371d92f4d2e4b"} Oct 30 12:14:25.707768 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:25.707696 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" podStartSLOduration=1.707680917 podStartE2EDuration="1.707680917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:25.705957492 +0000 UTC m=+1216.740095887" watchObservedRunningTime="2026-10-30 12:14:25.707680917 +0000 UTC m=+1216.741819312" Oct 30 12:14:25.747102 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:25.747016 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" podStartSLOduration=1.7470006169999999 podStartE2EDuration="1.747000617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:25.744685188 +0000 UTC m=+1216.778823583" watchObservedRunningTime="2026-10-30 12:14:25.747000617 +0000 UTC m=+1216.781139013" Oct 30 12:14:26.585541 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:26.585508 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" Oct 30 12:14:26.585719 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:26.585590 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" Oct 30 12:14:28.628938 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:28.628906 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" Oct 30 12:14:28.629479 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:28.628991 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" Oct 30 12:14:29.665265 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:29.665236 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" Oct 30 12:14:31.877720 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:31.877680 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79"] Oct 30 12:14:31.882886 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:31.882840 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" Oct 30 12:14:31.898514 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:31.898470 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-c1b1e8-serving-cert\"" Oct 30 12:14:31.935256 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:31.935147 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79"] Oct 30 12:14:31.954285 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:31.954242 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcddfa21-70dc-407b-803f-c88f7ae0d27e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79\" (UID: \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" Oct 30 12:14:31.954526 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:31.954326 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcddfa21-70dc-407b-803f-c88f7ae0d27e-metrics-certs\") pod \"maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79\" (UID: \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" Oct 30 12:14:31.954526 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:31.954452 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcddfa21-70dc-407b-803f-c88f7ae0d27e-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79\" (UID: \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" Oct 30 12:14:31.954657 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:31.954529 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7skz\" (UniqueName: \"kubernetes.io/projected/bcddfa21-70dc-407b-803f-c88f7ae0d27e-kube-api-access-g7skz\") pod \"maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79\" (UID: \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" Oct 30 12:14:32.056203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:32.056162 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcddfa21-70dc-407b-803f-c88f7ae0d27e-metrics-certs\") pod \"maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79\" (UID: \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" Oct 30 12:14:32.056402 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:32.056220 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcddfa21-70dc-407b-803f-c88f7ae0d27e-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79\" (UID: \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" Oct 30 12:14:32.056402 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:32.056268 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7skz\" (UniqueName: \"kubernetes.io/projected/bcddfa21-70dc-407b-803f-c88f7ae0d27e-kube-api-access-g7skz\") pod \"maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79\" (UID: \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" Oct 30 12:14:32.056402 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:32.056352 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcddfa21-70dc-407b-803f-c88f7ae0d27e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79\" (UID: \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" Oct 30 12:14:32.057036 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:32.057005 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcddfa21-70dc-407b-803f-c88f7ae0d27e-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79\" (UID: \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" Oct 30 12:14:32.058893 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:32.058864 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcddfa21-70dc-407b-803f-c88f7ae0d27e-metrics-certs\") pod \"maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79\" (UID: \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" Oct 30 12:14:32.059004 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:32.058900 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcddfa21-70dc-407b-803f-c88f7ae0d27e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79\" (UID: \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" Oct 30 12:14:32.079525 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:32.079494 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7skz\" (UniqueName: \"kubernetes.io/projected/bcddfa21-70dc-407b-803f-c88f7ae0d27e-kube-api-access-g7skz\") pod \"maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79\" (UID: \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" Oct 30 12:14:32.198739 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:32.198181 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" Oct 30 12:14:32.540020 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:32.539993 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79"] Oct 30 12:14:32.541607 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:14:32.541581 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcddfa21_70dc_407b_803f_c88f7ae0d27e.slice/crio-43e6dde9a1a0c198e9a2887138768c50a072cd0c2f2beb37856d8eba45d503f4 WatchSource:0}: Error finding container 43e6dde9a1a0c198e9a2887138768c50a072cd0c2f2beb37856d8eba45d503f4: Status 404 returned error can't find the container with id 43e6dde9a1a0c198e9a2887138768c50a072cd0c2f2beb37856d8eba45d503f4 Oct 30 12:14:32.543348 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:32.543320 4468 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:14:32.711882 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:32.711837 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" event={"ID":"bcddfa21-70dc-407b-803f-c88f7ae0d27e","Type":"ContainerStarted","Data":"43e6dde9a1a0c198e9a2887138768c50a072cd0c2f2beb37856d8eba45d503f4"} Oct 30 12:14:33.720736 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:33.720700 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" event={"ID":"bcddfa21-70dc-407b-803f-c88f7ae0d27e","Type":"ContainerStarted","Data":"5129c6cf22d0f5d84be956fe26cc114a6dd9e009ea12898c3d6297134a30d0f4"} Oct 30 12:14:33.721283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:33.720975 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" Oct 30 12:14:33.762995 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:33.762935 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" podStartSLOduration=2.355999675 podStartE2EDuration="2.762913674s" totalImagesPullingTime="406.913999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:33.760609367 +0000 UTC m=+1224.794747766" watchObservedRunningTime="2026-10-30 12:14:33.762913674 +0000 UTC m=+1224.797052069" Oct 30 12:14:36.696339 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:36.695265 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" Oct 30 12:14:36.702376 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:36.702351 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" Oct 30 12:14:36.755866 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:36.754876 4468 generic.go:356] "Generic (PLEG): container finished" podID="189793bd-8385-4741-85aa-3d89fc9a1d30" containerID="8bc6e9749f1b5d86d39134cea2c68e807cab5bcd4fbfeecd1d587e2919976070" exitCode=0 Oct 30 12:14:36.755866 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:36.755095 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" event={"ID":"189793bd-8385-4741-85aa-3d89fc9a1d30","Type":"ContainerDied","Data":"8bc6e9749f1b5d86d39134cea2c68e807cab5bcd4fbfeecd1d587e2919976070"} Oct 30 12:14:36.755866 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:36.755549 4468 scope.go:122] "RemoveContainer" containerID="8bc6e9749f1b5d86d39134cea2c68e807cab5bcd4fbfeecd1d587e2919976070" Oct 30 12:14:36.757310 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:36.757284 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:14:36.757513 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:36.757489 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:14:36.757599 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:36.757545 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:14:37.762271 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:37.762229 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" event={"ID":"189793bd-8385-4741-85aa-3d89fc9a1d30","Type":"ContainerStarted","Data":"df31c4131c9ad60d5b14a267527ba6c248b524f8001e68c61aec318a0e43e899"} Oct 30 12:14:38.190237 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:38.190196 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:38.191205 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:38.191179 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 12:14:38.191320 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:38.191229 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" podUID="189793bd-8385-4741-85aa-3d89fc9a1d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 12:14:39.190977 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:39.190930 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 12:14:39.191409 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:39.190994 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" podUID="189793bd-8385-4741-85aa-3d89fc9a1d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 12:14:39.755562 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:39.755515 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" Oct 30 12:14:40.191504 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:40.190303 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 12:14:40.191504 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:40.190378 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" podUID="189793bd-8385-4741-85aa-3d89fc9a1d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 12:14:41.190126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:41.189994 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 12:14:41.190126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:41.190060 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" podUID="189793bd-8385-4741-85aa-3d89fc9a1d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 12:14:42.190867 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:42.190825 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 12:14:42.191319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:42.190878 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" podUID="189793bd-8385-4741-85aa-3d89fc9a1d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 12:14:42.541650 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:42.541559 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:43.220590 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:43.220544 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:43.230115 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:43.230062 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:14:47.826389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:47.826354 4468 generic.go:356] "Generic (PLEG): container finished" podID="56010c34-d070-4709-8ec7-6c9a23178226" containerID="5eda8ae7d53781a7002538e18113ad4943607ce95403afeab63c01b68a12ba20" exitCode=0 Oct 30 12:14:47.826840 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:47.826436 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" event={"ID":"56010c34-d070-4709-8ec7-6c9a23178226","Type":"ContainerDied","Data":"5eda8ae7d53781a7002538e18113ad4943607ce95403afeab63c01b68a12ba20"} Oct 30 12:14:47.826879 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:47.826840 4468 scope.go:122] "RemoveContainer" containerID="5eda8ae7d53781a7002538e18113ad4943607ce95403afeab63c01b68a12ba20" Oct 30 12:14:47.828261 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:47.828235 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:14:47.828331 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:47.828302 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:14:47.828367 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:47.828332 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:14:48.836771 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:48.836735 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" event={"ID":"56010c34-d070-4709-8ec7-6c9a23178226","Type":"ContainerStarted","Data":"ca8a447a91f4011e46daba356df258ffc63f96360c98a49489c6d0c9278e1683"} Oct 30 12:14:49.528097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:49.527242 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:49.529321 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:49.529276 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 12:14:49.529466 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:49.529342 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" podUID="56010c34-d070-4709-8ec7-6c9a23178226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 12:14:50.523098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:50.517484 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 12:14:50.523098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:50.517551 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" podUID="56010c34-d070-4709-8ec7-6c9a23178226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 12:14:50.628245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:50.628195 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:51.522099 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:51.517565 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 12:14:51.522099 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:51.517686 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" podUID="56010c34-d070-4709-8ec7-6c9a23178226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 12:14:52.522094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:52.518497 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 12:14:52.522094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:52.518560 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" podUID="56010c34-d070-4709-8ec7-6c9a23178226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 12:14:53.522095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:53.521528 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 12:14:53.522095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:53.521612 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" podUID="56010c34-d070-4709-8ec7-6c9a23178226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 12:14:53.599837 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:14:53.599801 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:14:54.521854 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:54.521824 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:54.522836 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:54.522816 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:14:59.642915 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.642875 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5"] Oct 30 12:14:59.645836 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.645818 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:14:59.650998 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.650955 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-dshm\") pod \"premium-w4-512ec5-kserve-b784cb9cf-2xxr5\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:14:59.651196 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.651012 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-tmp-dir\") pod \"premium-w4-512ec5-kserve-b784cb9cf-2xxr5\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:14:59.651320 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.651291 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjr4f\" (UniqueName: \"kubernetes.io/projected/2ea5e911-0baf-41f3-8ccc-39aa997f4151-kube-api-access-zjr4f\") pod \"premium-w4-512ec5-kserve-b784cb9cf-2xxr5\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:14:59.651392 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.651347 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea5e911-0baf-41f3-8ccc-39aa997f4151-tls-certs\") pod \"premium-w4-512ec5-kserve-b784cb9cf-2xxr5\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:14:59.651392 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.651385 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-home\") pod \"premium-w4-512ec5-kserve-b784cb9cf-2xxr5\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:14:59.651504 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.651421 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-model-cache\") pod \"premium-w4-512ec5-kserve-b784cb9cf-2xxr5\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:14:59.676619 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.676578 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"kube-root-ca.crt\"" Oct 30 12:14:59.676957 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.676593 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"default-dockercfg-2qxls\"" Oct 30 12:14:59.677123 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.676632 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"openshift-service-ca.crt\"" Oct 30 12:14:59.677123 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.676662 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"premium-w4-512ec5-kserve-self-signed-certs\"" Oct 30 12:14:59.680710 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.680669 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5"] Oct 30 12:14:59.752590 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.752550 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjr4f\" (UniqueName: \"kubernetes.io/projected/2ea5e911-0baf-41f3-8ccc-39aa997f4151-kube-api-access-zjr4f\") pod \"premium-w4-512ec5-kserve-b784cb9cf-2xxr5\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:14:59.752590 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.752589 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea5e911-0baf-41f3-8ccc-39aa997f4151-tls-certs\") pod \"premium-w4-512ec5-kserve-b784cb9cf-2xxr5\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:14:59.752857 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.752616 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-home\") pod \"premium-w4-512ec5-kserve-b784cb9cf-2xxr5\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:14:59.752857 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.752640 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-model-cache\") pod \"premium-w4-512ec5-kserve-b784cb9cf-2xxr5\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:14:59.752857 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.752701 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-dshm\") pod \"premium-w4-512ec5-kserve-b784cb9cf-2xxr5\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:14:59.752857 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.752726 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-tmp-dir\") pod \"premium-w4-512ec5-kserve-b784cb9cf-2xxr5\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:14:59.753093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.753040 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-home\") pod \"premium-w4-512ec5-kserve-b784cb9cf-2xxr5\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:14:59.753169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.753144 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-model-cache\") pod \"premium-w4-512ec5-kserve-b784cb9cf-2xxr5\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:14:59.753206 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.753175 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-tmp-dir\") pod \"premium-w4-512ec5-kserve-b784cb9cf-2xxr5\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:14:59.755141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.755114 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-dshm\") pod \"premium-w4-512ec5-kserve-b784cb9cf-2xxr5\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:14:59.755338 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.755321 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea5e911-0baf-41f3-8ccc-39aa997f4151-tls-certs\") pod \"premium-w4-512ec5-kserve-b784cb9cf-2xxr5\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:14:59.775672 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.775635 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjr4f\" (UniqueName: \"kubernetes.io/projected/2ea5e911-0baf-41f3-8ccc-39aa997f4151-kube-api-access-zjr4f\") pod \"premium-w4-512ec5-kserve-b784cb9cf-2xxr5\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:14:59.959417 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:14:59.959372 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:15:00.169735 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.169697 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx"] Oct 30 12:15:00.172597 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.172568 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" Oct 30 12:15:00.183883 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.183851 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm"] Oct 30 12:15:00.186899 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.186870 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" Oct 30 12:15:00.192887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.192839 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:15:00.196924 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.196565 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx"] Oct 30 12:15:00.210430 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.210353 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm"] Oct 30 12:15:00.257027 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.256986 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32998321-e50b-4668-8021-48e81474c9bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm\" (UID: \"32998321-e50b-4668-8021-48e81474c9bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" Oct 30 12:15:00.257229 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.257034 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9beaf1-5b19-467b-aead-b8e3493d001a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889375-g7zrx\" (UID: \"3d9beaf1-5b19-467b-aead-b8e3493d001a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" Oct 30 12:15:00.257229 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.257175 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdqqr\" (UniqueName: \"kubernetes.io/projected/32998321-e50b-4668-8021-48e81474c9bf-kube-api-access-kdqqr\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm\" (UID: \"32998321-e50b-4668-8021-48e81474c9bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" Oct 30 12:15:00.257229 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.257225 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dvql\" (UniqueName: \"kubernetes.io/projected/3d9beaf1-5b19-467b-aead-b8e3493d001a-kube-api-access-4dvql\") pod \"maas-api-key-cleanup-29889375-g7zrx\" (UID: \"3d9beaf1-5b19-467b-aead-b8e3493d001a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" Oct 30 12:15:00.262597 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.262568 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5"] Oct 30 12:15:00.262697 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:15:00.262627 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ea5e911_0baf_41f3_8ccc_39aa997f4151.slice/crio-c4da4b45429675dd774aad3bebcd679cf9acfbe73259d87e33cbdb1e8fb8e4d6 WatchSource:0}: Error finding container c4da4b45429675dd774aad3bebcd679cf9acfbe73259d87e33cbdb1e8fb8e4d6: Status 404 returned error can't find the container with id c4da4b45429675dd774aad3bebcd679cf9acfbe73259d87e33cbdb1e8fb8e4d6 Oct 30 12:15:00.358438 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.358404 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32998321-e50b-4668-8021-48e81474c9bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm\" (UID: \"32998321-e50b-4668-8021-48e81474c9bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" Oct 30 12:15:00.358438 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.358447 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9beaf1-5b19-467b-aead-b8e3493d001a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889375-g7zrx\" (UID: \"3d9beaf1-5b19-467b-aead-b8e3493d001a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" Oct 30 12:15:00.358704 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.358501 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdqqr\" (UniqueName: \"kubernetes.io/projected/32998321-e50b-4668-8021-48e81474c9bf-kube-api-access-kdqqr\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm\" (UID: \"32998321-e50b-4668-8021-48e81474c9bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" Oct 30 12:15:00.358704 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.358639 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dvql\" (UniqueName: \"kubernetes.io/projected/3d9beaf1-5b19-467b-aead-b8e3493d001a-kube-api-access-4dvql\") pod \"maas-api-key-cleanup-29889375-g7zrx\" (UID: \"3d9beaf1-5b19-467b-aead-b8e3493d001a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" Oct 30 12:15:00.359278 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.359253 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9beaf1-5b19-467b-aead-b8e3493d001a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889375-g7zrx\" (UID: \"3d9beaf1-5b19-467b-aead-b8e3493d001a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" Oct 30 12:15:00.359424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.359305 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32998321-e50b-4668-8021-48e81474c9bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm\" (UID: \"32998321-e50b-4668-8021-48e81474c9bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" Oct 30 12:15:00.381870 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.381839 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dvql\" (UniqueName: \"kubernetes.io/projected/3d9beaf1-5b19-467b-aead-b8e3493d001a-kube-api-access-4dvql\") pod \"maas-api-key-cleanup-29889375-g7zrx\" (UID: \"3d9beaf1-5b19-467b-aead-b8e3493d001a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" Oct 30 12:15:00.382027 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.381946 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdqqr\" (UniqueName: \"kubernetes.io/projected/32998321-e50b-4668-8021-48e81474c9bf-kube-api-access-kdqqr\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm\" (UID: \"32998321-e50b-4668-8021-48e81474c9bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" Oct 30 12:15:00.486350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.486260 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" Oct 30 12:15:00.504885 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.504850 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" Oct 30 12:15:00.786943 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.786914 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx"] Oct 30 12:15:00.788448 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:15:00.788414 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9beaf1_5b19_467b_aead_b8e3493d001a.slice/crio-f703f0121c9fc3138eda8c37aea7c1e192dc35973add0184a999c0626792bd9d WatchSource:0}: Error finding container f703f0121c9fc3138eda8c37aea7c1e192dc35973add0184a999c0626792bd9d: Status 404 returned error can't find the container with id f703f0121c9fc3138eda8c37aea7c1e192dc35973add0184a999c0626792bd9d Oct 30 12:15:00.823980 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.823946 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm"] Oct 30 12:15:00.824428 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:15:00.824406 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32998321_e50b_4668_8021_48e81474c9bf.slice/crio-87f12b16fa1aefbf02dd13a0764dfa404bb1632a4f7b395e49c756f8d4afa0fb WatchSource:0}: Error finding container 87f12b16fa1aefbf02dd13a0764dfa404bb1632a4f7b395e49c756f8d4afa0fb: Status 404 returned error can't find the container with id 87f12b16fa1aefbf02dd13a0764dfa404bb1632a4f7b395e49c756f8d4afa0fb Oct 30 12:15:00.931479 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.931438 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" event={"ID":"3d9beaf1-5b19-467b-aead-b8e3493d001a","Type":"ContainerStarted","Data":"f703f0121c9fc3138eda8c37aea7c1e192dc35973add0184a999c0626792bd9d"} Oct 30 12:15:00.932761 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.932727 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" event={"ID":"32998321-e50b-4668-8021-48e81474c9bf","Type":"ContainerStarted","Data":"87f12b16fa1aefbf02dd13a0764dfa404bb1632a4f7b395e49c756f8d4afa0fb"} Oct 30 12:15:00.934404 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.934351 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" event={"ID":"2ea5e911-0baf-41f3-8ccc-39aa997f4151","Type":"ContainerStarted","Data":"147fc8b34348ec09b51ed85b02b2b901d76a5e693f9aee7de87c8384c41a916e"} Oct 30 12:15:00.934404 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.934387 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" event={"ID":"2ea5e911-0baf-41f3-8ccc-39aa997f4151","Type":"ContainerStarted","Data":"c4da4b45429675dd774aad3bebcd679cf9acfbe73259d87e33cbdb1e8fb8e4d6"} Oct 30 12:15:00.959633 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.959598 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:15:00.983145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:00.983091 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podStartSLOduration=1.98305917 podStartE2EDuration="1.98305917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:00.982053543 +0000 UTC m=+1252.016191941" watchObservedRunningTime="2026-10-30 12:15:00.98305917 +0000 UTC m=+1252.017197566" Oct 30 12:15:03.964987 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:03.964941 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" event={"ID":"3d9beaf1-5b19-467b-aead-b8e3493d001a","Type":"ContainerStarted","Data":"d4e399302c30f72a78ee084bcc3f1501d6173dbed5e150b094d0616224604313"} Oct 30 12:15:03.966874 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:03.966840 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" event={"ID":"32998321-e50b-4668-8021-48e81474c9bf","Type":"ContainerStarted","Data":"6b028e2f5a4843fd24f13a686b796f49e531d27934f803127a56e302ee2be8bc"} Oct 30 12:15:04.006666 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:04.006617 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" podStartSLOduration=1.564461873 podStartE2EDuration="4.006601961s" totalImagesPullingTime="2.442140088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:04.005505039 +0000 UTC m=+1255.039643434" watchObservedRunningTime="2026-10-30 12:15:04.006601961 +0000 UTC m=+1255.040740356" Oct 30 12:15:04.047191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:04.047134 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" podStartSLOduration=1.511859056 podStartE2EDuration="4.047119734s" totalImagesPullingTime="2.535260678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:04.045275432 +0000 UTC m=+1255.079413835" watchObservedRunningTime="2026-10-30 12:15:04.047119734 +0000 UTC m=+1255.081258129" Oct 30 12:15:04.974547 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:04.974512 4468 generic.go:356] "Generic (PLEG): container finished" podID="3d9beaf1-5b19-467b-aead-b8e3493d001a" containerID="d4e399302c30f72a78ee084bcc3f1501d6173dbed5e150b094d0616224604313" exitCode=60 Oct 30 12:15:04.975006 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:04.974604 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" event={"ID":"3d9beaf1-5b19-467b-aead-b8e3493d001a","Type":"ContainerDied","Data":"d4e399302c30f72a78ee084bcc3f1501d6173dbed5e150b094d0616224604313"} Oct 30 12:15:04.975006 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:04.974853 4468 scope.go:122] "RemoveContainer" containerID="d4e399302c30f72a78ee084bcc3f1501d6173dbed5e150b094d0616224604313" Oct 30 12:15:04.976523 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:04.976501 4468 generic.go:356] "Generic (PLEG): container finished" podID="32998321-e50b-4668-8021-48e81474c9bf" containerID="6b028e2f5a4843fd24f13a686b796f49e531d27934f803127a56e302ee2be8bc" exitCode=60 Oct 30 12:15:04.976629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:04.976532 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" event={"ID":"32998321-e50b-4668-8021-48e81474c9bf","Type":"ContainerDied","Data":"6b028e2f5a4843fd24f13a686b796f49e531d27934f803127a56e302ee2be8bc"} Oct 30 12:15:04.976884 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:04.976867 4468 scope.go:122] "RemoveContainer" containerID="6b028e2f5a4843fd24f13a686b796f49e531d27934f803127a56e302ee2be8bc" Oct 30 12:15:05.983665 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:05.983625 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" event={"ID":"3d9beaf1-5b19-467b-aead-b8e3493d001a","Type":"ContainerStarted","Data":"f3f7cf76c4dd9013ccc2292cc37c58f22fe9a53a0ba7b8eeb8db8bcec650c3aa"} Oct 30 12:15:05.985454 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:05.985423 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" event={"ID":"32998321-e50b-4668-8021-48e81474c9bf","Type":"ContainerStarted","Data":"1e3b4a17c447231d5e01a5e774914cdae85e2d3548b32a9ae74670f84d5e8818"} Oct 30 12:15:06.993851 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:06.993816 4468 generic.go:356] "Generic (PLEG): container finished" podID="3d9beaf1-5b19-467b-aead-b8e3493d001a" containerID="f3f7cf76c4dd9013ccc2292cc37c58f22fe9a53a0ba7b8eeb8db8bcec650c3aa" exitCode=60 Oct 30 12:15:06.994345 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:06.993901 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" event={"ID":"3d9beaf1-5b19-467b-aead-b8e3493d001a","Type":"ContainerDied","Data":"f3f7cf76c4dd9013ccc2292cc37c58f22fe9a53a0ba7b8eeb8db8bcec650c3aa"} Oct 30 12:15:06.994345 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:06.993969 4468 scope.go:122] "RemoveContainer" containerID="d4e399302c30f72a78ee084bcc3f1501d6173dbed5e150b094d0616224604313" Oct 30 12:15:06.994345 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:06.994178 4468 scope.go:122] "RemoveContainer" containerID="f3f7cf76c4dd9013ccc2292cc37c58f22fe9a53a0ba7b8eeb8db8bcec650c3aa" Oct 30 12:15:06.994515 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:15:06.994480 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889375-g7zrx_odh-ai-gateway-infra(3d9beaf1-5b19-467b-aead-b8e3493d001a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" podUID="3d9beaf1-5b19-467b-aead-b8e3493d001a" Oct 30 12:15:06.995839 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:06.995817 4468 generic.go:356] "Generic (PLEG): container finished" podID="32998321-e50b-4668-8021-48e81474c9bf" containerID="1e3b4a17c447231d5e01a5e774914cdae85e2d3548b32a9ae74670f84d5e8818" exitCode=60 Oct 30 12:15:06.995944 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:06.995902 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" event={"ID":"32998321-e50b-4668-8021-48e81474c9bf","Type":"ContainerDied","Data":"1e3b4a17c447231d5e01a5e774914cdae85e2d3548b32a9ae74670f84d5e8818"} Oct 30 12:15:06.996151 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:06.996136 4468 scope.go:122] "RemoveContainer" containerID="1e3b4a17c447231d5e01a5e774914cdae85e2d3548b32a9ae74670f84d5e8818" Oct 30 12:15:06.996391 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:15:06.996375 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm_odh-ai-gateway-infra(32998321-e50b-4668-8021-48e81474c9bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" podUID="32998321-e50b-4668-8021-48e81474c9bf" Oct 30 12:15:07.006029 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:07.006005 4468 scope.go:122] "RemoveContainer" containerID="6b028e2f5a4843fd24f13a686b796f49e531d27934f803127a56e302ee2be8bc" Oct 30 12:15:08.004079 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:08.004028 4468 scope.go:122] "RemoveContainer" containerID="f3f7cf76c4dd9013ccc2292cc37c58f22fe9a53a0ba7b8eeb8db8bcec650c3aa" Oct 30 12:15:08.004554 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:15:08.004393 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889375-g7zrx_odh-ai-gateway-infra(3d9beaf1-5b19-467b-aead-b8e3493d001a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" podUID="3d9beaf1-5b19-467b-aead-b8e3493d001a" Oct 30 12:15:08.005814 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:08.005789 4468 scope.go:122] "RemoveContainer" containerID="1e3b4a17c447231d5e01a5e774914cdae85e2d3548b32a9ae74670f84d5e8818" Oct 30 12:15:08.006099 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:15:08.006046 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm_odh-ai-gateway-infra(32998321-e50b-4668-8021-48e81474c9bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" podUID="32998321-e50b-4668-8021-48e81474c9bf" Oct 30 12:15:09.959993 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:09.959956 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:15:09.973625 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:09.973595 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:15:10.029723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:10.029694 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:15:17.082917 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:17.082875 4468 scope.go:122] "RemoveContainer" containerID="f3f7cf76c4dd9013ccc2292cc37c58f22fe9a53a0ba7b8eeb8db8bcec650c3aa" Oct 30 12:15:18.078346 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.078313 4468 generic.go:356] "Generic (PLEG): container finished" podID="3d9beaf1-5b19-467b-aead-b8e3493d001a" containerID="54af17f4d6c14b146467c1ae91e7d8c16248bb9f30cdf3b1260c2daf17ab12ea" exitCode=60 Oct 30 12:15:18.078866 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.078390 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" event={"ID":"3d9beaf1-5b19-467b-aead-b8e3493d001a","Type":"ContainerDied","Data":"54af17f4d6c14b146467c1ae91e7d8c16248bb9f30cdf3b1260c2daf17ab12ea"} Oct 30 12:15:18.078866 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.078454 4468 scope.go:122] "RemoveContainer" containerID="f3f7cf76c4dd9013ccc2292cc37c58f22fe9a53a0ba7b8eeb8db8bcec650c3aa" Oct 30 12:15:18.078991 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.078871 4468 scope.go:122] "RemoveContainer" containerID="54af17f4d6c14b146467c1ae91e7d8c16248bb9f30cdf3b1260c2daf17ab12ea" Oct 30 12:15:18.079192 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:15:18.079172 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889375-g7zrx_odh-ai-gateway-infra(3d9beaf1-5b19-467b-aead-b8e3493d001a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" podUID="3d9beaf1-5b19-467b-aead-b8e3493d001a" Oct 30 12:15:18.083304 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.083279 4468 scope.go:122] "RemoveContainer" containerID="1e3b4a17c447231d5e01a5e774914cdae85e2d3548b32a9ae74670f84d5e8818" Oct 30 12:15:18.655829 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.655734 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z"] Oct 30 12:15:18.658450 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.658425 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:18.678920 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.678883 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z"] Oct 30 12:15:18.687528 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.687495 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"default-dockercfg-px7j9\"" Oct 30 12:15:18.687697 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.687617 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"kube-root-ca.crt\"" Oct 30 12:15:18.687697 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.687631 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"openshift-service-ca.crt\"" Oct 30 12:15:18.687843 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.687741 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"premium-w0-f920f7-kserve-self-signed-certs\"" Oct 30 12:15:18.792384 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.792344 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxwjw\" (UniqueName: \"kubernetes.io/projected/fa877b54-11c9-4574-9d5e-e0d00dff0020-kube-api-access-kxwjw\") pod \"premium-w0-f920f7-kserve-c85dc7858-h5n2z\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:18.792384 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.792388 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-dshm\") pod \"premium-w0-f920f7-kserve-c85dc7858-h5n2z\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:18.792615 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.792404 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa877b54-11c9-4574-9d5e-e0d00dff0020-tls-certs\") pod \"premium-w0-f920f7-kserve-c85dc7858-h5n2z\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:18.792615 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.792475 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-model-cache\") pod \"premium-w0-f920f7-kserve-c85dc7858-h5n2z\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:18.792615 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.792584 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-home\") pod \"premium-w0-f920f7-kserve-c85dc7858-h5n2z\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:18.792739 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.792725 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-tmp-dir\") pod \"premium-w0-f920f7-kserve-c85dc7858-h5n2z\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:18.893984 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.893944 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-tmp-dir\") pod \"premium-w0-f920f7-kserve-c85dc7858-h5n2z\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:18.894203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.893992 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxwjw\" (UniqueName: \"kubernetes.io/projected/fa877b54-11c9-4574-9d5e-e0d00dff0020-kube-api-access-kxwjw\") pod \"premium-w0-f920f7-kserve-c85dc7858-h5n2z\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:18.894203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.894022 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-dshm\") pod \"premium-w0-f920f7-kserve-c85dc7858-h5n2z\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:18.894203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.894044 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa877b54-11c9-4574-9d5e-e0d00dff0020-tls-certs\") pod \"premium-w0-f920f7-kserve-c85dc7858-h5n2z\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:18.894203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.894082 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-model-cache\") pod \"premium-w0-f920f7-kserve-c85dc7858-h5n2z\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:18.894203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.894143 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-home\") pod \"premium-w0-f920f7-kserve-c85dc7858-h5n2z\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:18.894594 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.894567 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-model-cache\") pod \"premium-w0-f920f7-kserve-c85dc7858-h5n2z\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:18.894679 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.894594 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-home\") pod \"premium-w0-f920f7-kserve-c85dc7858-h5n2z\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:18.894679 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.894608 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-tmp-dir\") pod \"premium-w0-f920f7-kserve-c85dc7858-h5n2z\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:18.896492 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.896469 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-dshm\") pod \"premium-w0-f920f7-kserve-c85dc7858-h5n2z\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:18.896776 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.896757 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa877b54-11c9-4574-9d5e-e0d00dff0020-tls-certs\") pod \"premium-w0-f920f7-kserve-c85dc7858-h5n2z\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:18.921877 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.921846 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxwjw\" (UniqueName: \"kubernetes.io/projected/fa877b54-11c9-4574-9d5e-e0d00dff0020-kube-api-access-kxwjw\") pod \"premium-w0-f920f7-kserve-c85dc7858-h5n2z\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:18.973223 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:18.973180 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:19.093272 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:19.093228 4468 generic.go:356] "Generic (PLEG): container finished" podID="32998321-e50b-4668-8021-48e81474c9bf" containerID="16f372d1442df0fd4ff205eddfc2397e37f82e96e08b07851c58776093765d3a" exitCode=60 Oct 30 12:15:19.095000 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:19.094966 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" event={"ID":"32998321-e50b-4668-8021-48e81474c9bf","Type":"ContainerDied","Data":"16f372d1442df0fd4ff205eddfc2397e37f82e96e08b07851c58776093765d3a"} Oct 30 12:15:19.095132 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:19.095057 4468 scope.go:122] "RemoveContainer" containerID="1e3b4a17c447231d5e01a5e774914cdae85e2d3548b32a9ae74670f84d5e8818" Oct 30 12:15:19.095447 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:19.095426 4468 scope.go:122] "RemoveContainer" containerID="16f372d1442df0fd4ff205eddfc2397e37f82e96e08b07851c58776093765d3a" Oct 30 12:15:19.095766 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:15:19.095741 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm_odh-ai-gateway-infra(32998321-e50b-4668-8021-48e81474c9bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" podUID="32998321-e50b-4668-8021-48e81474c9bf" Oct 30 12:15:19.130054 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:19.130004 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx"] Oct 30 12:15:19.279417 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:19.279384 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z"] Oct 30 12:15:19.279753 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:15:19.279731 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa877b54_11c9_4574_9d5e_e0d00dff0020.slice/crio-811d744ec996fe861bb440980416de3893c267f35e8fc302053893cd1f27dc50 WatchSource:0}: Error finding container 811d744ec996fe861bb440980416de3893c267f35e8fc302053893cd1f27dc50: Status 404 returned error can't find the container with id 811d744ec996fe861bb440980416de3893c267f35e8fc302053893cd1f27dc50 Oct 30 12:15:19.332485 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:19.332458 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" Oct 30 12:15:19.399920 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:19.399882 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d9beaf1-5b19-467b-aead-b8e3493d001a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9beaf1-5b19-467b-aead-b8e3493d001a-maas-api-service-ca\") pod \"3d9beaf1-5b19-467b-aead-b8e3493d001a\" (UID: \"3d9beaf1-5b19-467b-aead-b8e3493d001a\") " Oct 30 12:15:19.400118 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:19.400038 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d9beaf1-5b19-467b-aead-b8e3493d001a-kube-api-access-4dvql\" (UniqueName: \"kubernetes.io/projected/3d9beaf1-5b19-467b-aead-b8e3493d001a-kube-api-access-4dvql\") pod \"3d9beaf1-5b19-467b-aead-b8e3493d001a\" (UID: \"3d9beaf1-5b19-467b-aead-b8e3493d001a\") " Oct 30 12:15:19.400395 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:19.400360 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9beaf1-5b19-467b-aead-b8e3493d001a-maas-api-service-ca" pod "3d9beaf1-5b19-467b-aead-b8e3493d001a" (UID: "3d9beaf1-5b19-467b-aead-b8e3493d001a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:15:19.402373 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:19.402341 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9beaf1-5b19-467b-aead-b8e3493d001a-kube-api-access-4dvql" pod "3d9beaf1-5b19-467b-aead-b8e3493d001a" (UID: "3d9beaf1-5b19-467b-aead-b8e3493d001a"). InnerVolumeSpecName "kube-api-access-4dvql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:15:19.501159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:19.501049 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9beaf1-5b19-467b-aead-b8e3493d001a-maas-api-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:15:19.501159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:19.501112 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dvql\" (UniqueName: \"kubernetes.io/projected/3d9beaf1-5b19-467b-aead-b8e3493d001a-kube-api-access-4dvql\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:15:20.101564 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:20.101529 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" Oct 30 12:15:20.102163 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:20.101566 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx" event={"ID":"3d9beaf1-5b19-467b-aead-b8e3493d001a","Type":"ContainerDied","Data":"f703f0121c9fc3138eda8c37aea7c1e192dc35973add0184a999c0626792bd9d"} Oct 30 12:15:20.102163 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:20.101628 4468 scope.go:122] "RemoveContainer" containerID="54af17f4d6c14b146467c1ae91e7d8c16248bb9f30cdf3b1260c2daf17ab12ea" Oct 30 12:15:20.106266 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:20.106232 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" event={"ID":"fa877b54-11c9-4574-9d5e-e0d00dff0020","Type":"ContainerStarted","Data":"df8ab2b5560db3f6e0e1870ee331bf326207a803695a2e9d7c422fc4e2f02fc2"} Oct 30 12:15:20.106406 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:20.106278 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" event={"ID":"fa877b54-11c9-4574-9d5e-e0d00dff0020","Type":"ContainerStarted","Data":"811d744ec996fe861bb440980416de3893c267f35e8fc302053893cd1f27dc50"} Oct 30 12:15:20.152191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:20.152153 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm"] Oct 30 12:15:20.164249 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:20.164196 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podStartSLOduration=2.164179096 podStartE2EDuration="2.164179096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:20.155977754 +0000 UTC m=+1271.190116150" watchObservedRunningTime="2026-10-30 12:15:20.164179096 +0000 UTC m=+1271.198317491" Oct 30 12:15:20.193730 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:20.193663 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx"] Oct 30 12:15:20.196629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:20.196589 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889375-g7zrx"] Oct 30 12:15:20.326265 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:20.326237 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" Oct 30 12:15:20.409516 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:20.409479 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32998321-e50b-4668-8021-48e81474c9bf-kube-api-access-kdqqr\" (UniqueName: \"kubernetes.io/projected/32998321-e50b-4668-8021-48e81474c9bf-kube-api-access-kdqqr\") pod \"32998321-e50b-4668-8021-48e81474c9bf\" (UID: \"32998321-e50b-4668-8021-48e81474c9bf\") " Oct 30 12:15:20.409738 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:20.409699 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32998321-e50b-4668-8021-48e81474c9bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32998321-e50b-4668-8021-48e81474c9bf-maas-api-service-ca\") pod \"32998321-e50b-4668-8021-48e81474c9bf\" (UID: \"32998321-e50b-4668-8021-48e81474c9bf\") " Oct 30 12:15:20.410217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:20.410182 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32998321-e50b-4668-8021-48e81474c9bf-maas-api-service-ca" pod "32998321-e50b-4668-8021-48e81474c9bf" (UID: "32998321-e50b-4668-8021-48e81474c9bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:15:20.412103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:20.412050 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32998321-e50b-4668-8021-48e81474c9bf-kube-api-access-kdqqr" pod "32998321-e50b-4668-8021-48e81474c9bf" (UID: "32998321-e50b-4668-8021-48e81474c9bf"). InnerVolumeSpecName "kube-api-access-kdqqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:15:20.511161 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:20.511115 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32998321-e50b-4668-8021-48e81474c9bf-maas-api-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:15:20.511161 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:20.511156 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdqqr\" (UniqueName: \"kubernetes.io/projected/32998321-e50b-4668-8021-48e81474c9bf-kube-api-access-kdqqr\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:15:20.973953 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:20.973914 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:21.088874 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:21.088819 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d9beaf1-5b19-467b-aead-b8e3493d001a" path="/var/lib/kubelet/pods/3d9beaf1-5b19-467b-aead-b8e3493d001a/volumes" Oct 30 12:15:21.113655 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:21.113610 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" event={"ID":"32998321-e50b-4668-8021-48e81474c9bf","Type":"ContainerDied","Data":"87f12b16fa1aefbf02dd13a0764dfa404bb1632a4f7b395e49c756f8d4afa0fb"} Oct 30 12:15:21.114181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:21.113684 4468 scope.go:122] "RemoveContainer" containerID="16f372d1442df0fd4ff205eddfc2397e37f82e96e08b07851c58776093765d3a" Oct 30 12:15:21.114181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:21.113621 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm" Oct 30 12:15:21.164649 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:21.164614 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm"] Oct 30 12:15:21.179301 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:21.179250 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889375-d5jpm"] Oct 30 12:15:23.089381 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:23.089343 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32998321-e50b-4668-8021-48e81474c9bf" path="/var/lib/kubelet/pods/32998321-e50b-4668-8021-48e81474c9bf/volumes" Oct 30 12:15:23.599032 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:15:23.598992 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:15:28.973804 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:28.973760 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:29.005109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:29.004383 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:29.184914 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:29.184868 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:15:35.633916 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.633870 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh"] Oct 30 12:15:35.634733 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.634709 4468 cpu_manager.go:414] "containerMap: removing container" podUID="32998321-e50b-4668-8021-48e81474c9bf" containerName="cleanup" Oct 30 12:15:35.634838 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.634733 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32998321-e50b-4668-8021-48e81474c9bf" containerName="cleanup" Oct 30 12:15:35.634838 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.634751 4468 cpu_manager.go:414] "containerMap: removing container" podUID="32998321-e50b-4668-8021-48e81474c9bf" containerName="cleanup" Oct 30 12:15:35.634838 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.634760 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32998321-e50b-4668-8021-48e81474c9bf" containerName="cleanup" Oct 30 12:15:35.634838 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.634779 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3d9beaf1-5b19-467b-aead-b8e3493d001a" containerName="cleanup" Oct 30 12:15:35.634838 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.634786 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d9beaf1-5b19-467b-aead-b8e3493d001a" containerName="cleanup" Oct 30 12:15:35.634838 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.634796 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3d9beaf1-5b19-467b-aead-b8e3493d001a" containerName="cleanup" Oct 30 12:15:35.634838 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.634803 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d9beaf1-5b19-467b-aead-b8e3493d001a" containerName="cleanup" Oct 30 12:15:35.635237 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.634945 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d9beaf1-5b19-467b-aead-b8e3493d001a" containerName="cleanup" Oct 30 12:15:35.635237 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.634960 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="32998321-e50b-4668-8021-48e81474c9bf" containerName="cleanup" Oct 30 12:15:35.635237 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.634970 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="32998321-e50b-4668-8021-48e81474c9bf" containerName="cleanup" Oct 30 12:15:35.635237 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.634979 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d9beaf1-5b19-467b-aead-b8e3493d001a" containerName="cleanup" Oct 30 12:15:35.635237 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.634987 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d9beaf1-5b19-467b-aead-b8e3493d001a" containerName="cleanup" Oct 30 12:15:35.642302 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.642271 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:35.656018 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.655983 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh"] Oct 30 12:15:35.663964 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.663933 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"default-dockercfg-2vbn8\"" Oct 30 12:15:35.668374 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.668309 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"kube-root-ca.crt\"" Oct 30 12:15:35.668374 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.668314 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"premium-w1-c1b1e8-kserve-self-signed-certs\"" Oct 30 12:15:35.668374 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.668377 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"openshift-service-ca.crt\"" Oct 30 12:15:35.722556 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.722512 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-model-cache\") pod \"premium-w1-c1b1e8-kserve-cc8778c7-b8rkh\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:35.722763 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.722574 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-tls-certs\") pod \"premium-w1-c1b1e8-kserve-cc8778c7-b8rkh\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:35.722763 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.722675 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c6cr\" (UniqueName: \"kubernetes.io/projected/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-kube-api-access-5c6cr\") pod \"premium-w1-c1b1e8-kserve-cc8778c7-b8rkh\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:35.722863 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.722773 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-dshm\") pod \"premium-w1-c1b1e8-kserve-cc8778c7-b8rkh\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:35.722905 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.722862 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-tmp-dir\") pod \"premium-w1-c1b1e8-kserve-cc8778c7-b8rkh\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:35.722946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.722931 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-home\") pod \"premium-w1-c1b1e8-kserve-cc8778c7-b8rkh\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:35.824284 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.824244 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-tmp-dir\") pod \"premium-w1-c1b1e8-kserve-cc8778c7-b8rkh\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:35.824284 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.824288 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-home\") pod \"premium-w1-c1b1e8-kserve-cc8778c7-b8rkh\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:35.824568 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.824344 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-model-cache\") pod \"premium-w1-c1b1e8-kserve-cc8778c7-b8rkh\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:35.824568 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.824365 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-tls-certs\") pod \"premium-w1-c1b1e8-kserve-cc8778c7-b8rkh\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:35.824568 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.824392 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c6cr\" (UniqueName: \"kubernetes.io/projected/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-kube-api-access-5c6cr\") pod \"premium-w1-c1b1e8-kserve-cc8778c7-b8rkh\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:35.824568 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.824454 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-dshm\") pod \"premium-w1-c1b1e8-kserve-cc8778c7-b8rkh\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:35.824789 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.824745 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-tmp-dir\") pod \"premium-w1-c1b1e8-kserve-cc8778c7-b8rkh\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:35.824864 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.824801 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-home\") pod \"premium-w1-c1b1e8-kserve-cc8778c7-b8rkh\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:35.824864 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.824814 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-model-cache\") pod \"premium-w1-c1b1e8-kserve-cc8778c7-b8rkh\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:35.827120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.827085 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-dshm\") pod \"premium-w1-c1b1e8-kserve-cc8778c7-b8rkh\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:35.829190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.829161 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-tls-certs\") pod \"premium-w1-c1b1e8-kserve-cc8778c7-b8rkh\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:35.847547 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.847513 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c6cr\" (UniqueName: \"kubernetes.io/projected/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-kube-api-access-5c6cr\") pod \"premium-w1-c1b1e8-kserve-cc8778c7-b8rkh\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:35.961092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:35.960300 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:36.298442 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:15:36.298409 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea95fef4_f1e3_4967_9c5f_b6133ae05eb1.slice/crio-c72f400583e05574e20c661a55377ffeea535d247006881a184cfdb851ccee49 WatchSource:0}: Error finding container c72f400583e05574e20c661a55377ffeea535d247006881a184cfdb851ccee49: Status 404 returned error can't find the container with id c72f400583e05574e20c661a55377ffeea535d247006881a184cfdb851ccee49 Oct 30 12:15:36.298553 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:36.298528 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh"] Oct 30 12:15:37.230476 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:37.230427 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" event={"ID":"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1","Type":"ContainerStarted","Data":"668f6202dfac84c3c3345cf7bb340c8c0cb6df8f2bf89574e00bf3d8721ef881"} Oct 30 12:15:37.231005 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:37.230485 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" event={"ID":"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1","Type":"ContainerStarted","Data":"c72f400583e05574e20c661a55377ffeea535d247006881a184cfdb851ccee49"} Oct 30 12:15:37.270644 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:37.270592 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podStartSLOduration=2.270577518 podStartE2EDuration="2.270577518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:37.269831082 +0000 UTC m=+1288.303969478" watchObservedRunningTime="2026-10-30 12:15:37.270577518 +0000 UTC m=+1288.304715914" Oct 30 12:15:37.961284 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:37.961245 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:45.961280 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:45.961236 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:45.974466 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:45.974438 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:46.300788 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:46.300702 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:15:51.039147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.038900 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb"] Oct 30 12:15:51.039671 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.039453 4468 cpu_manager.go:414] "containerMap: removing container" podUID="32998321-e50b-4668-8021-48e81474c9bf" containerName="cleanup" Oct 30 12:15:51.039671 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.039467 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32998321-e50b-4668-8021-48e81474c9bf" containerName="cleanup" Oct 30 12:15:51.039671 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.039479 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3d9beaf1-5b19-467b-aead-b8e3493d001a" containerName="cleanup" Oct 30 12:15:51.039671 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.039485 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d9beaf1-5b19-467b-aead-b8e3493d001a" containerName="cleanup" Oct 30 12:15:51.039671 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.039591 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="32998321-e50b-4668-8021-48e81474c9bf" containerName="cleanup" Oct 30 12:15:51.044122 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.044094 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.061860 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.061827 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-512ec5-openshift-default-dockercfg-ww4j8\"" Oct 30 12:15:51.075362 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.075327 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb"] Oct 30 12:15:51.121984 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.121950 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxzmr\" (UniqueName: \"kubernetes.io/projected/e1a20ae7-502b-447e-ae58-120744dd79fa-kube-api-access-fxzmr\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.122188 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.122005 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.122188 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.122121 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.122188 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.122147 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.122188 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.122169 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.122404 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.122326 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.122404 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.122379 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1a20ae7-502b-447e-ae58-120744dd79fa-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.122487 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.122473 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.122527 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.122504 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.223465 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.223428 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.223637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.223487 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.223637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.223517 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.223778 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.223626 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.223842 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.223770 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.223842 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.223819 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1a20ae7-502b-447e-ae58-120744dd79fa-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.223991 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.223965 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.224051 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.224021 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.224155 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.224133 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.224216 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.224135 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxzmr\" (UniqueName: \"kubernetes.io/projected/e1a20ae7-502b-447e-ae58-120744dd79fa-kube-api-access-fxzmr\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.224473 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.224453 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1a20ae7-502b-447e-ae58-120744dd79fa-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.224624 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.224600 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.224878 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.224854 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.225292 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.225266 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.226884 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.226841 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.227307 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.227286 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.247006 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.246965 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.252100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.248168 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxzmr\" (UniqueName: \"kubernetes.io/projected/e1a20ae7-502b-447e-ae58-120744dd79fa-kube-api-access-fxzmr\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.372442 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.372349 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:51.705239 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.704504 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb"] Oct 30 12:15:51.711220 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:15:51.711191 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a20ae7_502b_447e_ae58_120744dd79fa.slice/crio-6c5b602b3b19ef35719bc39ad04bb26d4799ce2db202201cc0bdf34451aa9d21 WatchSource:0}: Error finding container 6c5b602b3b19ef35719bc39ad04bb26d4799ce2db202201cc0bdf34451aa9d21: Status 404 returned error can't find the container with id 6c5b602b3b19ef35719bc39ad04bb26d4799ce2db202201cc0bdf34451aa9d21 Oct 30 12:15:51.717817 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.716274 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:15:51.717817 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.716369 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:15:51.717817 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:51.716412 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:15:52.328992 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:52.328953 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" event={"ID":"e1a20ae7-502b-447e-ae58-120744dd79fa","Type":"ContainerStarted","Data":"5ab9aa8b29a34505d45a1d36317486f74ab18388e30ec25c8863d37e4a5dcfd8"} Oct 30 12:15:52.329424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:52.329005 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" event={"ID":"e1a20ae7-502b-447e-ae58-120744dd79fa","Type":"ContainerStarted","Data":"6c5b602b3b19ef35719bc39ad04bb26d4799ce2db202201cc0bdf34451aa9d21"} Oct 30 12:15:52.372680 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:52.372638 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:52.374429 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:52.374403 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 12:15:52.374551 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:52.374453 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" podUID="e1a20ae7-502b-447e-ae58-120744dd79fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 12:15:52.389257 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:52.389193 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" podStartSLOduration=1.389177237 podStartE2EDuration="1.389177237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:52.384045436 +0000 UTC m=+1303.418183831" watchObservedRunningTime="2026-10-30 12:15:52.389177237 +0000 UTC m=+1303.423315632" Oct 30 12:15:53.373799 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:53.373764 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 12:15:53.374216 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:53.373822 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" podUID="e1a20ae7-502b-447e-ae58-120744dd79fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 12:15:53.600966 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:15:53.600925 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:15:54.373861 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:54.373820 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 12:15:54.374315 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:54.373891 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" podUID="e1a20ae7-502b-447e-ae58-120744dd79fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 12:15:55.372880 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:55.372830 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 12:15:55.373128 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:55.372910 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" podUID="e1a20ae7-502b-447e-ae58-120744dd79fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 12:15:56.373561 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:56.373521 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 12:15:56.373999 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:56.373592 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" podUID="e1a20ae7-502b-447e-ae58-120744dd79fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 12:15:57.373166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:57.373132 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 12:15:57.373353 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:57.373197 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" podUID="e1a20ae7-502b-447e-ae58-120744dd79fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 12:15:58.372833 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:58.372796 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 12:15:58.373290 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:58.372868 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" podUID="e1a20ae7-502b-447e-ae58-120744dd79fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 12:15:59.377463 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:59.377427 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:59.377884 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:59.377756 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:15:59.378690 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:15:59.378670 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 12:16:23.599883 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:16:23.599845 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:16:24.169297 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.169262 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6"] Oct 30 12:16:24.174550 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.174524 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.188658 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.188629 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-47fd99-openshift-default-dockercfg-lmqzd\"" Oct 30 12:16:24.204123 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.204060 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6"] Oct 30 12:16:24.299420 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.299381 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.299420 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.299417 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.299658 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.299535 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.299658 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.299595 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.299658 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.299625 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.299658 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.299650 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.299813 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.299681 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qv6z\" (UniqueName: \"kubernetes.io/projected/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-kube-api-access-7qv6z\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.299813 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.299797 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.299890 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.299836 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.401327 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.401282 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.401327 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.401324 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.401568 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.401378 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.401568 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.401424 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.401568 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.401464 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.401568 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.401506 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.401568 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.401533 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qv6z\" (UniqueName: \"kubernetes.io/projected/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-kube-api-access-7qv6z\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.401832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.401622 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.401832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.401667 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.401832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.401779 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.401832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.401801 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.401832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.401830 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.402026 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.402008 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.402306 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.402285 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.404027 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.403998 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.404337 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.404301 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.424818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.424741 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.425036 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.424832 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qv6z\" (UniqueName: \"kubernetes.io/projected/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-kube-api-access-7qv6z\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.488121 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.488048 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:24.809690 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.809657 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6"] Oct 30 12:16:24.810041 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:16:24.810018 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b6677b_eb87_4662_9b1b_2c0a4832d1d5.slice/crio-13496f895be400ea4178652cf12b2206c029494773c4f69288b817c99100eb59 WatchSource:0}: Error finding container 13496f895be400ea4178652cf12b2206c029494773c4f69288b817c99100eb59: Status 404 returned error can't find the container with id 13496f895be400ea4178652cf12b2206c029494773c4f69288b817c99100eb59 Oct 30 12:16:24.812374 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.812338 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:16:24.812472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.812417 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:16:24.812472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:24.812448 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:16:25.538049 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:25.538012 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" event={"ID":"29b6677b-eb87-4662-9b1b-2c0a4832d1d5","Type":"ContainerStarted","Data":"394b075ee687c46161d0f994ab4d474aff633fceb5cc2d632bc009192ba2560a"} Oct 30 12:16:25.538249 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:25.538086 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" event={"ID":"29b6677b-eb87-4662-9b1b-2c0a4832d1d5","Type":"ContainerStarted","Data":"13496f895be400ea4178652cf12b2206c029494773c4f69288b817c99100eb59"} Oct 30 12:16:25.595742 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:25.595683 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" podStartSLOduration=1.5956659499999999 podStartE2EDuration="1.59566595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:16:25.5940381 +0000 UTC m=+1336.628176498" watchObservedRunningTime="2026-10-30 12:16:25.59566595 +0000 UTC m=+1336.629804410" Oct 30 12:16:26.488630 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:26.488579 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:26.490396 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:26.490369 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 12:16:26.490519 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:26.490424 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" podUID="29b6677b-eb87-4662-9b1b-2c0a4832d1d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 12:16:27.489249 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:27.489217 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 12:16:27.489684 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:27.489270 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" podUID="29b6677b-eb87-4662-9b1b-2c0a4832d1d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 12:16:28.488606 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:28.488568 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 12:16:28.488805 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:28.488640 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" podUID="29b6677b-eb87-4662-9b1b-2c0a4832d1d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 12:16:29.488412 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:29.488377 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 12:16:29.488861 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:29.488426 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" podUID="29b6677b-eb87-4662-9b1b-2c0a4832d1d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 12:16:30.488536 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:30.488499 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 12:16:30.489056 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:30.488564 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" podUID="29b6677b-eb87-4662-9b1b-2c0a4832d1d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 12:16:32.263812 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:32.263776 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:32.264307 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:32.264200 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:32.265888 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:16:32.265858 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:16:53.602279 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:16:53.602239 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:17:10.305780 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.305743 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq"] Oct 30 12:17:10.309806 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.309776 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.331845 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.331816 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq"] Oct 30 12:17:10.340834 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.340793 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"kube-root-ca.crt\"" Oct 30 12:17:10.341057 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.340852 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"openshift-service-ca.crt\"" Oct 30 12:17:10.341057 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.341010 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"distinct-w2-47fd99-kserve-self-signed-certs\"" Oct 30 12:17:10.341057 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.341034 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"default-dockercfg-tkn4k\"" Oct 30 12:17:10.459277 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.459240 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bs9n\" (UniqueName: \"kubernetes.io/projected/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-kube-api-access-9bs9n\") pod \"distinct-w2-47fd99-kserve-66b6c4d966-8t5tq\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.459476 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.459295 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-tls-certs\") pod \"distinct-w2-47fd99-kserve-66b6c4d966-8t5tq\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.459476 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.459398 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-home\") pod \"distinct-w2-47fd99-kserve-66b6c4d966-8t5tq\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.459476 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.459439 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-tmp-dir\") pod \"distinct-w2-47fd99-kserve-66b6c4d966-8t5tq\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.459651 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.459622 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-dshm\") pod \"distinct-w2-47fd99-kserve-66b6c4d966-8t5tq\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.459714 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.459696 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-model-cache\") pod \"distinct-w2-47fd99-kserve-66b6c4d966-8t5tq\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.564145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.564036 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-model-cache\") pod \"distinct-w2-47fd99-kserve-66b6c4d966-8t5tq\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.564355 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.564167 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bs9n\" (UniqueName: \"kubernetes.io/projected/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-kube-api-access-9bs9n\") pod \"distinct-w2-47fd99-kserve-66b6c4d966-8t5tq\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.564355 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.564208 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-tls-certs\") pod \"distinct-w2-47fd99-kserve-66b6c4d966-8t5tq\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.564355 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.564267 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-home\") pod \"distinct-w2-47fd99-kserve-66b6c4d966-8t5tq\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.564355 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.564315 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-tmp-dir\") pod \"distinct-w2-47fd99-kserve-66b6c4d966-8t5tq\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.564610 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.564419 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-dshm\") pod \"distinct-w2-47fd99-kserve-66b6c4d966-8t5tq\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.566609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.566576 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-home\") pod \"distinct-w2-47fd99-kserve-66b6c4d966-8t5tq\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.566785 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.566718 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-model-cache\") pod \"distinct-w2-47fd99-kserve-66b6c4d966-8t5tq\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.567100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.567048 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-tmp-dir\") pod \"distinct-w2-47fd99-kserve-66b6c4d966-8t5tq\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.568011 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.567986 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-dshm\") pod \"distinct-w2-47fd99-kserve-66b6c4d966-8t5tq\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.569758 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.569737 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-tls-certs\") pod \"distinct-w2-47fd99-kserve-66b6c4d966-8t5tq\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.589100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.589054 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bs9n\" (UniqueName: \"kubernetes.io/projected/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-kube-api-access-9bs9n\") pod \"distinct-w2-47fd99-kserve-66b6c4d966-8t5tq\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.629381 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.629342 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:10.934800 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:17:10.934755 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76e7b4b9_e1bf_4cd7_9137_1108398cc50e.slice/crio-0d56d5b77ff8aba2535618e6cbc6c2b28b724616fd03f70eec333acae7a4b79c WatchSource:0}: Error finding container 0d56d5b77ff8aba2535618e6cbc6c2b28b724616fd03f70eec333acae7a4b79c: Status 404 returned error can't find the container with id 0d56d5b77ff8aba2535618e6cbc6c2b28b724616fd03f70eec333acae7a4b79c Oct 30 12:17:10.934920 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:10.934810 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq"] Oct 30 12:17:11.884165 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:11.884117 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" event={"ID":"76e7b4b9-e1bf-4cd7-9137-1108398cc50e","Type":"ContainerStarted","Data":"535f0611367523074cbbff213269aa991b138aa3e3b8078e9e10375037f26231"} Oct 30 12:17:11.884165 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:11.884174 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" event={"ID":"76e7b4b9-e1bf-4cd7-9137-1108398cc50e","Type":"ContainerStarted","Data":"0d56d5b77ff8aba2535618e6cbc6c2b28b724616fd03f70eec333acae7a4b79c"} Oct 30 12:17:11.925686 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:11.925630 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podStartSLOduration=1.925615147 podStartE2EDuration="1.925615147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:11.924353445 +0000 UTC m=+1382.958491841" watchObservedRunningTime="2026-10-30 12:17:11.925615147 +0000 UTC m=+1382.959753542" Oct 30 12:17:12.629561 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:12.629523 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:19.758196 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:19.758143 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6"] Oct 30 12:17:19.763138 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:19.763110 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:19.791547 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:19.791511 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"distinct-2-w0-f920f7-kserve-self-signed-certs\"" Oct 30 12:17:19.794276 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:19.794242 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6"] Oct 30 12:17:19.892641 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:19.892604 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-dshm\") pod \"distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:19.892641 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:19.892648 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-home\") pod \"distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:19.892846 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:19.892668 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-model-cache\") pod \"distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:19.892846 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:19.892784 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1903b7f6-f2e3-4000-9399-a7117583640f-tls-certs\") pod \"distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:19.892846 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:19.892817 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv4ds\" (UniqueName: \"kubernetes.io/projected/1903b7f6-f2e3-4000-9399-a7117583640f-kube-api-access-lv4ds\") pod \"distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:19.892984 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:19.892901 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-tmp-dir\") pod \"distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:19.995725 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:19.995674 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-dshm\") pod \"distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:19.996539 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:19.996511 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-home\") pod \"distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:19.997110 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:19.997092 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-model-cache\") pod \"distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:19.997590 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:19.997563 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1903b7f6-f2e3-4000-9399-a7117583640f-tls-certs\") pod \"distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:19.997707 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:19.997620 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv4ds\" (UniqueName: \"kubernetes.io/projected/1903b7f6-f2e3-4000-9399-a7117583640f-kube-api-access-lv4ds\") pod \"distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:19.997777 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:19.997708 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-tmp-dir\") pod \"distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:19.997837 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:19.997472 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-model-cache\") pod \"distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:19.997837 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:19.997022 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-home\") pod \"distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:19.998301 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:19.998276 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-tmp-dir\") pod \"distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:20.000648 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:20.000623 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-dshm\") pod \"distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:20.001317 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:20.001294 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1903b7f6-f2e3-4000-9399-a7117583640f-tls-certs\") pod \"distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:20.053144 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:20.047585 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv4ds\" (UniqueName: \"kubernetes.io/projected/1903b7f6-f2e3-4000-9399-a7117583640f-kube-api-access-lv4ds\") pod \"distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:20.077275 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:20.077244 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:20.381587 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:20.381551 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6"] Oct 30 12:17:20.381732 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:17:20.381708 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1903b7f6_f2e3_4000_9399_a7117583640f.slice/crio-4d5c7a3bc4589f1331a295d364fd75716b1c9dcae746ecd06a7e1f9389982072 WatchSource:0}: Error finding container 4d5c7a3bc4589f1331a295d364fd75716b1c9dcae746ecd06a7e1f9389982072: Status 404 returned error can't find the container with id 4d5c7a3bc4589f1331a295d364fd75716b1c9dcae746ecd06a7e1f9389982072 Oct 30 12:17:20.629727 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:20.629621 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:20.643445 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:20.643414 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:20.947630 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:20.947589 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" event={"ID":"1903b7f6-f2e3-4000-9399-a7117583640f","Type":"ContainerStarted","Data":"3590f8f7699f08cafcb0942f859a4e8ad1b113d1a58238fb72ce65edcf0c777d"} Oct 30 12:17:20.947630 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:20.947639 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" event={"ID":"1903b7f6-f2e3-4000-9399-a7117583640f","Type":"ContainerStarted","Data":"4d5c7a3bc4589f1331a295d364fd75716b1c9dcae746ecd06a7e1f9389982072"} Oct 30 12:17:20.959599 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:20.959569 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:17:20.988857 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:20.988804 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podStartSLOduration=1.988788046 podStartE2EDuration="1.988788046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:20.987565568 +0000 UTC m=+1392.021703966" watchObservedRunningTime="2026-10-30 12:17:20.988788046 +0000 UTC m=+1392.022926441" Oct 30 12:17:21.077637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:21.077596 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:23.599258 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:17:23.599216 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:17:30.077755 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:30.077697 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:30.091034 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:30.090998 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:31.021964 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:31.021932 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:17:34.556797 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.556756 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz"] Oct 30 12:17:34.561232 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.561203 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:34.578435 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.578403 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz"] Oct 30 12:17:34.586948 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.586915 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"distinct-2-w1-c1b1e8-kserve-self-signed-certs\"" Oct 30 12:17:34.757403 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.756823 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-dshm\") pod \"distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:34.757403 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.756892 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d273864b-7bab-4471-8531-3855be5b6182-tls-certs\") pod \"distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:34.757403 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.756936 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-model-cache\") pod \"distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:34.757403 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.757144 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv5tq\" (UniqueName: \"kubernetes.io/projected/d273864b-7bab-4471-8531-3855be5b6182-kube-api-access-zv5tq\") pod \"distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:34.757403 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.757183 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-home\") pod \"distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:34.757403 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.757228 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-tmp-dir\") pod \"distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:34.858653 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.858611 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-model-cache\") pod \"distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:34.858828 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.858743 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv5tq\" (UniqueName: \"kubernetes.io/projected/d273864b-7bab-4471-8531-3855be5b6182-kube-api-access-zv5tq\") pod \"distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:34.858828 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.858776 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-home\") pod \"distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:34.858828 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.858805 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-tmp-dir\") pod \"distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:34.858997 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.858865 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-dshm\") pod \"distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:34.858997 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.858888 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d273864b-7bab-4471-8531-3855be5b6182-tls-certs\") pod \"distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:34.859129 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.859051 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-model-cache\") pod \"distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:34.859190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.859170 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-home\") pod \"distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:34.859325 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.859300 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-tmp-dir\") pod \"distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:34.861195 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.861168 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-dshm\") pod \"distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:34.861360 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.861342 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d273864b-7bab-4471-8531-3855be5b6182-tls-certs\") pod \"distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:34.881470 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:34.881431 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv5tq\" (UniqueName: \"kubernetes.io/projected/d273864b-7bab-4471-8531-3855be5b6182-kube-api-access-zv5tq\") pod \"distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:35.175601 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:35.175561 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:35.479252 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:17:35.479219 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd273864b_7bab_4471_8531_3855be5b6182.slice/crio-1b6ce39e5bcd272057ae0e234ec28adbc962b7a7b04b49aca07d1ba3e9628737 WatchSource:0}: Error finding container 1b6ce39e5bcd272057ae0e234ec28adbc962b7a7b04b49aca07d1ba3e9628737: Status 404 returned error can't find the container with id 1b6ce39e5bcd272057ae0e234ec28adbc962b7a7b04b49aca07d1ba3e9628737 Oct 30 12:17:35.479417 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:35.479250 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz"] Oct 30 12:17:36.064125 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.064078 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" event={"ID":"d273864b-7bab-4471-8531-3855be5b6182","Type":"ContainerStarted","Data":"99769e6d7b4aa20d868c87df93106ccb2e498389923bf38f94c73be30e5064f0"} Oct 30 12:17:36.064540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.064133 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" event={"ID":"d273864b-7bab-4471-8531-3855be5b6182","Type":"ContainerStarted","Data":"1b6ce39e5bcd272057ae0e234ec28adbc962b7a7b04b49aca07d1ba3e9628737"} Oct 30 12:17:36.105963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.105911 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podStartSLOduration=2.105896413 podStartE2EDuration="2.105896413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:36.104236907 +0000 UTC m=+1407.138375306" watchObservedRunningTime="2026-10-30 12:17:36.105896413 +0000 UTC m=+1407.140034809" Oct 30 12:17:36.175688 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.175646 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:36.329135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.329035 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm"] Oct 30 12:17:36.333330 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.333305 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.355462 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.355429 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm"] Oct 30 12:17:36.479347 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.479307 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.479531 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.479358 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d37dfa55-ca61-425e-b32e-7b9846eadd70-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.479531 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.479385 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.479628 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.479551 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqskj\" (UniqueName: \"kubernetes.io/projected/d37dfa55-ca61-425e-b32e-7b9846eadd70-kube-api-access-gqskj\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.479686 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.479668 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.479747 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.479729 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.479864 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.479849 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.479957 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.479933 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.480032 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.480014 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.581751 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.581647 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.581751 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.581716 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.582009 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.581754 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d37dfa55-ca61-425e-b32e-7b9846eadd70-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.582009 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.581790 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.582009 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.581832 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqskj\" (UniqueName: \"kubernetes.io/projected/d37dfa55-ca61-425e-b32e-7b9846eadd70-kube-api-access-gqskj\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.582009 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.581891 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.582009 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.581935 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.582331 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.582017 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.582331 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.582056 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.582441 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.582357 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.582441 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.582419 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.582537 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.582469 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.582592 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.582547 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.582675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.582645 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d37dfa55-ca61-425e-b32e-7b9846eadd70-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.585000 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.584977 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.585359 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.585337 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.604261 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.604228 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.604472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.604446 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqskj\" (UniqueName: \"kubernetes.io/projected/d37dfa55-ca61-425e-b32e-7b9846eadd70-kube-api-access-gqskj\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.648421 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.648390 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:36.992102 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.992056 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm"] Oct 30 12:17:36.992603 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:17:36.992570 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37dfa55_ca61_425e_b32e_7b9846eadd70.slice/crio-51c629a5e30fe67b8a54f857abd38a697f8aec5389057f2e294fc23af9b4b63e WatchSource:0}: Error finding container 51c629a5e30fe67b8a54f857abd38a697f8aec5389057f2e294fc23af9b4b63e: Status 404 returned error can't find the container with id 51c629a5e30fe67b8a54f857abd38a697f8aec5389057f2e294fc23af9b4b63e Oct 30 12:17:36.995437 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.995402 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:36.995541 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.995489 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:36.995609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:36.995549 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:37.073052 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:37.073023 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" event={"ID":"d37dfa55-ca61-425e-b32e-7b9846eadd70","Type":"ContainerStarted","Data":"51c629a5e30fe67b8a54f857abd38a697f8aec5389057f2e294fc23af9b4b63e"} Oct 30 12:17:38.081298 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:38.081251 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" event={"ID":"d37dfa55-ca61-425e-b32e-7b9846eadd70","Type":"ContainerStarted","Data":"bcd8a9707e902fc7f4d5588bc1a5226306420768daab801f9d9d2d6dbf19af56"} Oct 30 12:17:38.155926 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:38.155872 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" podStartSLOduration=2.155855938 podStartE2EDuration="2.155855938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:38.14214099 +0000 UTC m=+1409.176279386" watchObservedRunningTime="2026-10-30 12:17:38.155855938 +0000 UTC m=+1409.189994396" Oct 30 12:17:38.654306 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:38.654265 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:38.656329 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:38.656299 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 12:17:38.656461 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:38.656357 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" podUID="d37dfa55-ca61-425e-b32e-7b9846eadd70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 12:17:39.385814 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.385775 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn"] Oct 30 12:17:39.389907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.389884 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.433963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.433916 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn"] Oct 30 12:17:39.541801 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.541754 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x55ql\" (UniqueName: \"kubernetes.io/projected/32eeefc6-4a6c-439a-b825-fc318449dc66-kube-api-access-x55ql\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.542098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.542061 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.542233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.542209 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32eeefc6-4a6c-439a-b825-fc318449dc66-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.542353 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.542245 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.542353 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.542307 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.542353 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.542345 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.542502 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.542475 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.542609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.542555 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.542676 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.542625 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.643918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.643811 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x55ql\" (UniqueName: \"kubernetes.io/projected/32eeefc6-4a6c-439a-b825-fc318449dc66-kube-api-access-x55ql\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.643918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.643876 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.643918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.643915 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32eeefc6-4a6c-439a-b825-fc318449dc66-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.644234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.643947 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.644234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.643978 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.644234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.644013 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.644234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.644130 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.644234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.644208 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.644470 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.644299 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.644592 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.644568 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.644640 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.644595 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.644676 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.644636 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.644754 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.644734 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.644823 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.644805 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32eeefc6-4a6c-439a-b825-fc318449dc66-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.646394 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.646374 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.646618 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.646600 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.650195 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.650168 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 12:17:39.650392 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.650226 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" podUID="d37dfa55-ca61-425e-b32e-7b9846eadd70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 12:17:39.666349 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.666315 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.666527 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.666422 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x55ql\" (UniqueName: \"kubernetes.io/projected/32eeefc6-4a6c-439a-b825-fc318449dc66-kube-api-access-x55ql\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:39.724096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:39.723309 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:40.055138 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:40.055107 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn"] Oct 30 12:17:40.055348 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:17:40.055324 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32eeefc6_4a6c_439a_b825_fc318449dc66.slice/crio-b49e61ac0593b487d78631ff1e27591e415e52ac8c200478a12767a453150ce7 WatchSource:0}: Error finding container b49e61ac0593b487d78631ff1e27591e415e52ac8c200478a12767a453150ce7: Status 404 returned error can't find the container with id b49e61ac0593b487d78631ff1e27591e415e52ac8c200478a12767a453150ce7 Oct 30 12:17:40.057654 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:40.057618 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:40.057755 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:40.057698 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:40.057755 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:40.057744 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:40.100197 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:40.100108 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" event={"ID":"32eeefc6-4a6c-439a-b825-fc318449dc66","Type":"ContainerStarted","Data":"b49e61ac0593b487d78631ff1e27591e415e52ac8c200478a12767a453150ce7"} Oct 30 12:17:40.651819 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:40.651773 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 12:17:40.652424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:40.651844 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" podUID="d37dfa55-ca61-425e-b32e-7b9846eadd70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 12:17:41.136095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:41.135304 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" event={"ID":"32eeefc6-4a6c-439a-b825-fc318449dc66","Type":"ContainerStarted","Data":"4b472ff45a71f0849d987df82a0a77be300887d40a7996850261cb7a14a1ae84"} Oct 30 12:17:41.232237 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:41.232169 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" podStartSLOduration=2.232146785 podStartE2EDuration="2.232146785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:41.199492421 +0000 UTC m=+1412.233630807" watchObservedRunningTime="2026-10-30 12:17:41.232146785 +0000 UTC m=+1412.266285183" Oct 30 12:17:41.653095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:41.649516 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 12:17:41.653095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:41.649602 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" podUID="d37dfa55-ca61-425e-b32e-7b9846eadd70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 12:17:41.728095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:41.727331 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:41.728095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:41.727477 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 12:17:41.728095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:41.727522 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" podUID="32eeefc6-4a6c-439a-b825-fc318449dc66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 12:17:42.650094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:42.649555 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 12:17:42.650094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:42.649622 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" podUID="d37dfa55-ca61-425e-b32e-7b9846eadd70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 12:17:42.730093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:42.729561 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 12:17:42.730093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:42.729644 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" podUID="32eeefc6-4a6c-439a-b825-fc318449dc66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 12:17:43.649811 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:43.649772 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 12:17:43.649991 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:43.649843 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" podUID="d37dfa55-ca61-425e-b32e-7b9846eadd70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 12:17:43.729135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:43.728576 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 12:17:43.729135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:43.728657 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" podUID="32eeefc6-4a6c-439a-b825-fc318449dc66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 12:17:44.737199 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:44.736314 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 12:17:44.737199 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:44.736391 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" podUID="32eeefc6-4a6c-439a-b825-fc318449dc66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 12:17:45.175880 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:45.175849 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:45.190243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:45.190215 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:45.226095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:45.222401 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:17:45.692300 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:45.692253 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:17:45.692534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:45.692356 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" podUID="d37dfa55-ca61-425e-b32e-7b9846eadd70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:17:45.737096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:45.734540 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 12:17:45.737096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:45.734625 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" podUID="32eeefc6-4a6c-439a-b825-fc318449dc66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 12:17:46.170533 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:46.170498 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:46.170922 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:46.170898 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:46.171768 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:46.171747 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:17:46.726080 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:46.726019 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 12:17:46.726265 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:46.726123 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" podUID="32eeefc6-4a6c-439a-b825-fc318449dc66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 12:17:47.771569 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:47.767722 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:47.771569 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:47.768994 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:47.778913 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:47.778885 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:17:53.600273 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:17:53.600230 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:17:55.569858 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.569823 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk"] Oct 30 12:17:55.575496 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.575465 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.612113 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.612078 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk"] Oct 30 12:17:55.735818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.735773 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.736012 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.735836 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.736012 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.735977 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.736169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.736020 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfzgc\" (UniqueName: \"kubernetes.io/projected/64cf635f-868a-4021-91aa-f36a314e26c1-kube-api-access-vfzgc\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.736169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.736049 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64cf635f-868a-4021-91aa-f36a314e26c1-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.736240 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.736169 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.736277 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.736246 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.736277 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.736266 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64cf635f-868a-4021-91aa-f36a314e26c1-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.736353 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.736308 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64cf635f-868a-4021-91aa-f36a314e26c1-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.837731 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.837635 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.837731 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.837714 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.837965 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.837740 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64cf635f-868a-4021-91aa-f36a314e26c1-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.837965 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.837798 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64cf635f-868a-4021-91aa-f36a314e26c1-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.837965 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.837828 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.838197 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.838173 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.838275 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.838208 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.838374 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.838358 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.838701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.838651 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.838701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.838670 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64cf635f-868a-4021-91aa-f36a314e26c1-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.838872 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.838747 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.838872 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.838821 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfzgc\" (UniqueName: \"kubernetes.io/projected/64cf635f-868a-4021-91aa-f36a314e26c1-kube-api-access-vfzgc\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.838872 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.838864 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64cf635f-868a-4021-91aa-f36a314e26c1-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.839379 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.839352 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.840996 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.840971 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64cf635f-868a-4021-91aa-f36a314e26c1-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.842290 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.842267 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.893093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.890183 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64cf635f-868a-4021-91aa-f36a314e26c1-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.893093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.890823 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfzgc\" (UniqueName: \"kubernetes.io/projected/64cf635f-868a-4021-91aa-f36a314e26c1-kube-api-access-vfzgc\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:55.908349 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:55.908313 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:56.238278 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:56.237281 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks"] Oct 30 12:17:56.238278 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:56.237544 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" podUID="278f2976-5db1-4683-a372-01d103a69b92" containerName="istio-proxy" containerID="cri-o://4828811eda867e3467712d277d943c05bbbd6827ad0f4fc1c82c7b0aca1ea52f" gracePeriod=30 Oct 30 12:17:56.250451 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:56.250409 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t"] Oct 30 12:17:56.250952 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:56.250916 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" podUID="20bde431-93de-47ab-9cb6-fe90d1a4730a" containerName="istio-proxy" containerID="cri-o://8f1ff5ce81835ec05895069ad6bb73488748542ae8fb7831918bf417428acbdc" gracePeriod=30 Oct 30 12:17:56.356565 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:56.356500 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk"] Oct 30 12:17:56.380665 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:17:56.380620 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64cf635f_868a_4021_91aa_f36a314e26c1.slice/crio-0054c83a779898c0b1bf125e8aeca0daa3f2bc2eecc6c9527a4e045ab5afe915 WatchSource:0}: Error finding container 0054c83a779898c0b1bf125e8aeca0daa3f2bc2eecc6c9527a4e045ab5afe915: Status 404 returned error can't find the container with id 0054c83a779898c0b1bf125e8aeca0daa3f2bc2eecc6c9527a4e045ab5afe915 Oct 30 12:17:56.383581 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:56.383110 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:56.383581 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:56.383197 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:56.383581 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:56.383246 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:57.287636 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:57.287600 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" event={"ID":"64cf635f-868a-4021-91aa-f36a314e26c1","Type":"ContainerStarted","Data":"fdca0830c58d7f99db6c6c6b0c2ffe1aab4f3312d7d6d0804897d1d59a434a65"} Oct 30 12:17:57.288088 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:57.287654 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" event={"ID":"64cf635f-868a-4021-91aa-f36a314e26c1","Type":"ContainerStarted","Data":"0054c83a779898c0b1bf125e8aeca0daa3f2bc2eecc6c9527a4e045ab5afe915"} Oct 30 12:17:57.348762 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:57.348681 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" podStartSLOduration=2.348660907 podStartE2EDuration="2.348660907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:57.346160417 +0000 UTC m=+1428.380298815" watchObservedRunningTime="2026-10-30 12:17:57.348660907 +0000 UTC m=+1428.382799304" Oct 30 12:17:57.910095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:57.909448 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:17:57.929317 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:57.929258 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 12:17:57.929542 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:57.929352 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" podUID="64cf635f-868a-4021-91aa-f36a314e26c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 12:17:58.923094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:58.918492 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 12:17:58.923094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:58.918560 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" podUID="64cf635f-868a-4021-91aa-f36a314e26c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 12:17:59.920093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:59.915644 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 12:17:59.920093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:17:59.915742 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" podUID="64cf635f-868a-4021-91aa-f36a314e26c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 12:18:00.912108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:00.910504 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 12:18:00.912108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:00.910588 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" podUID="64cf635f-868a-4021-91aa-f36a314e26c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 12:18:01.574570 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.574518 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:18:01.594577 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.594516 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278f2976-5db1-4683-a372-01d103a69b92-istio-token\" (UniqueName: \"kubernetes.io/projected/278f2976-5db1-4683-a372-01d103a69b92-istio-token\") pod \"278f2976-5db1-4683-a372-01d103a69b92\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " Oct 30 12:18:01.594577 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.594582 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/278f2976-5db1-4683-a372-01d103a69b92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278f2976-5db1-4683-a372-01d103a69b92-istiod-ca-cert\") pod \"278f2976-5db1-4683-a372-01d103a69b92\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " Oct 30 12:18:01.594829 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.594643 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278f2976-5db1-4683-a372-01d103a69b92-kube-api-access-x8hvc\" (UniqueName: \"kubernetes.io/projected/278f2976-5db1-4683-a372-01d103a69b92-kube-api-access-x8hvc\") pod \"278f2976-5db1-4683-a372-01d103a69b92\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " Oct 30 12:18:01.594829 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.594719 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-workload-certs\") pod \"278f2976-5db1-4683-a372-01d103a69b92\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " Oct 30 12:18:01.594829 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.594777 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/278f2976-5db1-4683-a372-01d103a69b92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278f2976-5db1-4683-a372-01d103a69b92-istio-podinfo\") pod \"278f2976-5db1-4683-a372-01d103a69b92\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " Oct 30 12:18:01.595297 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.595017 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-workload-certs" pod "278f2976-5db1-4683-a372-01d103a69b92" (UID: "278f2976-5db1-4683-a372-01d103a69b92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:18:01.595297 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.595030 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/278f2976-5db1-4683-a372-01d103a69b92-istiod-ca-cert" pod "278f2976-5db1-4683-a372-01d103a69b92" (UID: "278f2976-5db1-4683-a372-01d103a69b92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:18:01.595297 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.595131 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-istio-envoy\") pod \"278f2976-5db1-4683-a372-01d103a69b92\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " Oct 30 12:18:01.595297 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.595174 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-workload-socket\") pod \"278f2976-5db1-4683-a372-01d103a69b92\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " Oct 30 12:18:01.595297 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.595226 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-istio-data\") pod \"278f2976-5db1-4683-a372-01d103a69b92\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " Oct 30 12:18:01.595586 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.595312 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-credential-socket\") pod \"278f2976-5db1-4683-a372-01d103a69b92\" (UID: \"278f2976-5db1-4683-a372-01d103a69b92\") " Oct 30 12:18:01.596101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.595855 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278f2976-5db1-4683-a372-01d103a69b92-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:18:01.596239 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.596137 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:18:01.596239 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.595957 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-workload-socket" pod "278f2976-5db1-4683-a372-01d103a69b92" (UID: "278f2976-5db1-4683-a372-01d103a69b92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:18:01.596239 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.596174 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-istio-data" pod "278f2976-5db1-4683-a372-01d103a69b92" (UID: "278f2976-5db1-4683-a372-01d103a69b92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:18:01.596562 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.596537 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-credential-socket" pod "278f2976-5db1-4683-a372-01d103a69b92" (UID: "278f2976-5db1-4683-a372-01d103a69b92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:18:01.599367 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.599341 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/278f2976-5db1-4683-a372-01d103a69b92-istio-podinfo" pod "278f2976-5db1-4683-a372-01d103a69b92" (UID: "278f2976-5db1-4683-a372-01d103a69b92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:18:01.600251 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.600214 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278f2976-5db1-4683-a372-01d103a69b92-kube-api-access-x8hvc" pod "278f2976-5db1-4683-a372-01d103a69b92" (UID: "278f2976-5db1-4683-a372-01d103a69b92"). InnerVolumeSpecName "kube-api-access-x8hvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:18:01.600432 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.600407 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278f2976-5db1-4683-a372-01d103a69b92-istio-token" pod "278f2976-5db1-4683-a372-01d103a69b92" (UID: "278f2976-5db1-4683-a372-01d103a69b92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:18:01.602478 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.602445 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-istio-envoy" pod "278f2976-5db1-4683-a372-01d103a69b92" (UID: "278f2976-5db1-4683-a372-01d103a69b92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:18:01.665628 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.665596 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:18:01.697042 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.697005 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-podinfo\") pod \"20bde431-93de-47ab-9cb6-fe90d1a4730a\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " Oct 30 12:18:01.697238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.697078 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-workload-certs\") pod \"20bde431-93de-47ab-9cb6-fe90d1a4730a\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " Oct 30 12:18:01.697238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.697120 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-data\") pod \"20bde431-93de-47ab-9cb6-fe90d1a4730a\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " Oct 30 12:18:01.697238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.697149 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-credential-socket\") pod \"20bde431-93de-47ab-9cb6-fe90d1a4730a\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " Oct 30 12:18:01.697360 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.697257 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-envoy\") pod \"20bde431-93de-47ab-9cb6-fe90d1a4730a\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " Oct 30 12:18:01.697360 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.697290 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-token\" (UniqueName: \"kubernetes.io/projected/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-token\") pod \"20bde431-93de-47ab-9cb6-fe90d1a4730a\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " Oct 30 12:18:01.697360 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.697329 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-workload-socket\") pod \"20bde431-93de-47ab-9cb6-fe90d1a4730a\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " Oct 30 12:18:01.697503 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.697358 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20bde431-93de-47ab-9cb6-fe90d1a4730a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20bde431-93de-47ab-9cb6-fe90d1a4730a-istiod-ca-cert\") pod \"20bde431-93de-47ab-9cb6-fe90d1a4730a\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " Oct 30 12:18:01.697503 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.697369 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-workload-certs" pod "20bde431-93de-47ab-9cb6-fe90d1a4730a" (UID: "20bde431-93de-47ab-9cb6-fe90d1a4730a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:18:01.697503 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.697377 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20bde431-93de-47ab-9cb6-fe90d1a4730a-kube-api-access-rclns\" (UniqueName: \"kubernetes.io/projected/20bde431-93de-47ab-9cb6-fe90d1a4730a-kube-api-access-rclns\") pod \"20bde431-93de-47ab-9cb6-fe90d1a4730a\" (UID: \"20bde431-93de-47ab-9cb6-fe90d1a4730a\") " Oct 30 12:18:01.697882 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.697856 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-data" pod "20bde431-93de-47ab-9cb6-fe90d1a4730a" (UID: "20bde431-93de-47ab-9cb6-fe90d1a4730a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:18:01.698017 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.697995 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-credential-socket" pod "20bde431-93de-47ab-9cb6-fe90d1a4730a" (UID: "20bde431-93de-47ab-9cb6-fe90d1a4730a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:18:01.698113 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.698086 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-workload-socket" pod "20bde431-93de-47ab-9cb6-fe90d1a4730a" (UID: "20bde431-93de-47ab-9cb6-fe90d1a4730a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:18:01.698180 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.698113 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:18:01.698180 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.698131 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:18:01.698180 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.698147 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:18:01.698180 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.698160 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278f2976-5db1-4683-a372-01d103a69b92-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:18:01.698180 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.698172 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278f2976-5db1-4683-a372-01d103a69b92-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:18:01.698368 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.698186 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8hvc\" (UniqueName: \"kubernetes.io/projected/278f2976-5db1-4683-a372-01d103a69b92-kube-api-access-x8hvc\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:18:01.698368 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.698198 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:18:01.698368 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.698210 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20bde431-93de-47ab-9cb6-fe90d1a4730a-istiod-ca-cert" pod "20bde431-93de-47ab-9cb6-fe90d1a4730a" (UID: "20bde431-93de-47ab-9cb6-fe90d1a4730a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:18:01.698368 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.698218 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278f2976-5db1-4683-a372-01d103a69b92-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:18:01.699391 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.699373 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20bde431-93de-47ab-9cb6-fe90d1a4730a-kube-api-access-rclns" pod "20bde431-93de-47ab-9cb6-fe90d1a4730a" (UID: "20bde431-93de-47ab-9cb6-fe90d1a4730a"). InnerVolumeSpecName "kube-api-access-rclns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:18:01.699711 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.699680 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-podinfo" pod "20bde431-93de-47ab-9cb6-fe90d1a4730a" (UID: "20bde431-93de-47ab-9cb6-fe90d1a4730a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:18:01.699801 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.699781 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-token" pod "20bde431-93de-47ab-9cb6-fe90d1a4730a" (UID: "20bde431-93de-47ab-9cb6-fe90d1a4730a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:18:01.700019 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.699994 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-envoy" pod "20bde431-93de-47ab-9cb6-fe90d1a4730a" (UID: "20bde431-93de-47ab-9cb6-fe90d1a4730a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:18:01.799030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.798983 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:18:01.799030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.799021 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:18:01.799030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.799035 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20bde431-93de-47ab-9cb6-fe90d1a4730a-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:18:01.799325 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.799046 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rclns\" (UniqueName: \"kubernetes.io/projected/20bde431-93de-47ab-9cb6-fe90d1a4730a-kube-api-access-rclns\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:18:01.799325 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.799060 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:18:01.799325 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.799103 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:18:01.799325 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.799116 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:18:01.799325 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.799126 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20bde431-93de-47ab-9cb6-fe90d1a4730a-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:18:01.909736 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.909697 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 12:18:01.909981 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:01.909953 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" podUID="64cf635f-868a-4021-91aa-f36a314e26c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 12:18:02.354449 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.354348 4468 generic.go:356] "Generic (PLEG): container finished" podID="278f2976-5db1-4683-a372-01d103a69b92" containerID="4828811eda867e3467712d277d943c05bbbd6827ad0f4fc1c82c7b0aca1ea52f" exitCode=0 Oct 30 12:18:02.354449 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.354381 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" event={"ID":"278f2976-5db1-4683-a372-01d103a69b92","Type":"ContainerDied","Data":"4828811eda867e3467712d277d943c05bbbd6827ad0f4fc1c82c7b0aca1ea52f"} Oct 30 12:18:02.354449 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.354440 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" Oct 30 12:18:02.354973 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.354474 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks" event={"ID":"278f2976-5db1-4683-a372-01d103a69b92","Type":"ContainerDied","Data":"26ae04cb39013a617543828b15775094e09e1bf5287a50b3d83fec6259854997"} Oct 30 12:18:02.354973 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.354507 4468 scope.go:122] "RemoveContainer" containerID="4828811eda867e3467712d277d943c05bbbd6827ad0f4fc1c82c7b0aca1ea52f" Oct 30 12:18:02.356271 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.356245 4468 generic.go:356] "Generic (PLEG): container finished" podID="20bde431-93de-47ab-9cb6-fe90d1a4730a" containerID="8f1ff5ce81835ec05895069ad6bb73488748542ae8fb7831918bf417428acbdc" exitCode=0 Oct 30 12:18:02.356364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.356282 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" event={"ID":"20bde431-93de-47ab-9cb6-fe90d1a4730a","Type":"ContainerDied","Data":"8f1ff5ce81835ec05895069ad6bb73488748542ae8fb7831918bf417428acbdc"} Oct 30 12:18:02.356364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.356334 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" event={"ID":"20bde431-93de-47ab-9cb6-fe90d1a4730a","Type":"ContainerDied","Data":"fa918b70667379893ef329d4a55256c5ef6b30d0b1d1d3c93763648f414775bd"} Oct 30 12:18:02.356610 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.356584 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t" Oct 30 12:18:02.370915 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.370897 4468 scope.go:122] "RemoveContainer" containerID="4828811eda867e3467712d277d943c05bbbd6827ad0f4fc1c82c7b0aca1ea52f" Oct 30 12:18:02.371285 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:18:02.371261 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4828811eda867e3467712d277d943c05bbbd6827ad0f4fc1c82c7b0aca1ea52f\": container with ID starting with 4828811eda867e3467712d277d943c05bbbd6827ad0f4fc1c82c7b0aca1ea52f not found: ID does not exist" containerID="4828811eda867e3467712d277d943c05bbbd6827ad0f4fc1c82c7b0aca1ea52f" Oct 30 12:18:02.371397 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.371294 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4828811eda867e3467712d277d943c05bbbd6827ad0f4fc1c82c7b0aca1ea52f"} err="failed to get container status \"4828811eda867e3467712d277d943c05bbbd6827ad0f4fc1c82c7b0aca1ea52f\": rpc error: code = NotFound desc = could not find container \"4828811eda867e3467712d277d943c05bbbd6827ad0f4fc1c82c7b0aca1ea52f\": container with ID starting with 4828811eda867e3467712d277d943c05bbbd6827ad0f4fc1c82c7b0aca1ea52f not found: ID does not exist" Oct 30 12:18:02.371397 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.371313 4468 scope.go:122] "RemoveContainer" containerID="8f1ff5ce81835ec05895069ad6bb73488748542ae8fb7831918bf417428acbdc" Oct 30 12:18:02.381441 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.381417 4468 scope.go:122] "RemoveContainer" containerID="8f1ff5ce81835ec05895069ad6bb73488748542ae8fb7831918bf417428acbdc" Oct 30 12:18:02.381795 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:18:02.381770 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f1ff5ce81835ec05895069ad6bb73488748542ae8fb7831918bf417428acbdc\": container with ID starting with 8f1ff5ce81835ec05895069ad6bb73488748542ae8fb7831918bf417428acbdc not found: ID does not exist" containerID="8f1ff5ce81835ec05895069ad6bb73488748542ae8fb7831918bf417428acbdc" Oct 30 12:18:02.381903 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.381802 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f1ff5ce81835ec05895069ad6bb73488748542ae8fb7831918bf417428acbdc"} err="failed to get container status \"8f1ff5ce81835ec05895069ad6bb73488748542ae8fb7831918bf417428acbdc\": rpc error: code = NotFound desc = could not find container \"8f1ff5ce81835ec05895069ad6bb73488748542ae8fb7831918bf417428acbdc\": container with ID starting with 8f1ff5ce81835ec05895069ad6bb73488748542ae8fb7831918bf417428acbdc not found: ID does not exist" Oct 30 12:18:02.428122 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.422794 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks"] Oct 30 12:18:02.432245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.432214 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-sbxks"] Oct 30 12:18:02.464585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.464550 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t"] Oct 30 12:18:02.469029 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.468993 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-66j8t"] Oct 30 12:18:02.909544 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.909510 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 12:18:02.909707 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:02.909565 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" podUID="64cf635f-868a-4021-91aa-f36a314e26c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 12:18:03.090318 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:03.090276 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20bde431-93de-47ab-9cb6-fe90d1a4730a" path="/var/lib/kubelet/pods/20bde431-93de-47ab-9cb6-fe90d1a4730a/volumes" Oct 30 12:18:03.091044 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:03.091021 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="278f2976-5db1-4683-a372-01d103a69b92" path="/var/lib/kubelet/pods/278f2976-5db1-4683-a372-01d103a69b92/volumes" Oct 30 12:18:03.914848 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:03.914819 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:18:03.915396 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:03.915206 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:18:03.915937 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:03.915919 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 12:18:10.040233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.040192 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5"] Oct 30 12:18:10.040790 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.040776 4468 cpu_manager.go:414] "containerMap: removing container" podUID="20bde431-93de-47ab-9cb6-fe90d1a4730a" containerName="istio-proxy" Oct 30 12:18:10.040790 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.040790 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20bde431-93de-47ab-9cb6-fe90d1a4730a" containerName="istio-proxy" Oct 30 12:18:10.040924 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.040801 4468 cpu_manager.go:414] "containerMap: removing container" podUID="278f2976-5db1-4683-a372-01d103a69b92" containerName="istio-proxy" Oct 30 12:18:10.040924 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.040807 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="278f2976-5db1-4683-a372-01d103a69b92" containerName="istio-proxy" Oct 30 12:18:10.040924 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.040895 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="20bde431-93de-47ab-9cb6-fe90d1a4730a" containerName="istio-proxy" Oct 30 12:18:10.040924 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.040904 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="278f2976-5db1-4683-a372-01d103a69b92" containerName="istio-proxy" Oct 30 12:18:10.046053 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.046023 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.072383 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.069261 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5"] Oct 30 12:18:10.072383 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.071693 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"distinct-w4-512ec5-kserve-self-signed-certs\"" Oct 30 12:18:10.123195 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.123156 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147053ac-cddc-4e20-ba45-ef81e5d3f719-tls-certs\") pod \"distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.123195 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.123198 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvdnj\" (UniqueName: \"kubernetes.io/projected/147053ac-cddc-4e20-ba45-ef81e5d3f719-kube-api-access-cvdnj\") pod \"distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.123408 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.123322 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-tmp-dir\") pod \"distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.123463 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.123442 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-dshm\") pod \"distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.123533 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.123521 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-model-cache\") pod \"distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.123623 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.123610 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-home\") pod \"distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.241370 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.228827 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-dshm\") pod \"distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.241370 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.228981 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-model-cache\") pod \"distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.241370 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.229040 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-home\") pod \"distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.241370 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.229121 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147053ac-cddc-4e20-ba45-ef81e5d3f719-tls-certs\") pod \"distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.241370 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.229149 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvdnj\" (UniqueName: \"kubernetes.io/projected/147053ac-cddc-4e20-ba45-ef81e5d3f719-kube-api-access-cvdnj\") pod \"distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.241370 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.229222 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-tmp-dir\") pod \"distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.241370 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.229710 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-tmp-dir\") pod \"distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.241370 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.234632 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-model-cache\") pod \"distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.241370 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.234898 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-home\") pod \"distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.262092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.247701 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147053ac-cddc-4e20-ba45-ef81e5d3f719-tls-certs\") pod \"distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.262092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.260657 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-dshm\") pod \"distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.266093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.264792 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvdnj\" (UniqueName: \"kubernetes.io/projected/147053ac-cddc-4e20-ba45-ef81e5d3f719-kube-api-access-cvdnj\") pod \"distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.362438 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.362349 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:10.669265 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:10.669234 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5"] Oct 30 12:18:10.670002 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:18:10.669972 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod147053ac_cddc_4e20_ba45_ef81e5d3f719.slice/crio-a3f09e1b896586cd3ba0cff57c23f02ea6d0657e5212759660204306c77ee3c0 WatchSource:0}: Error finding container a3f09e1b896586cd3ba0cff57c23f02ea6d0657e5212759660204306c77ee3c0: Status 404 returned error can't find the container with id a3f09e1b896586cd3ba0cff57c23f02ea6d0657e5212759660204306c77ee3c0 Oct 30 12:18:11.416732 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:11.416684 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" event={"ID":"147053ac-cddc-4e20-ba45-ef81e5d3f719","Type":"ContainerStarted","Data":"ce78c872f3b60f0861af939896132f50f3dde93346456fa2c2943fa2681d8535"} Oct 30 12:18:11.417260 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:11.416740 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" event={"ID":"147053ac-cddc-4e20-ba45-ef81e5d3f719","Type":"ContainerStarted","Data":"a3f09e1b896586cd3ba0cff57c23f02ea6d0657e5212759660204306c77ee3c0"} Oct 30 12:18:11.464783 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:11.464724 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podStartSLOduration=1.4647062690000001 podStartE2EDuration="1.464706269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:11.46314841 +0000 UTC m=+1442.497286807" watchObservedRunningTime="2026-10-30 12:18:11.464706269 +0000 UTC m=+1442.498844665" Oct 30 12:18:12.363488 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:12.363450 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:20.363497 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:20.363452 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:20.376777 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:20.376746 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:20.524720 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:20.524686 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:23.613680 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:18:23.613639 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:18:24.529680 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.529639 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z"] Oct 30 12:18:24.534944 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.534917 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.565661 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.565623 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z"] Oct 30 12:18:24.651193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.651161 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2407a56-45e8-4257-b271-437daa98e3f4-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.651193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.651198 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.651699 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.651227 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.651699 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.651308 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vkdx\" (UniqueName: \"kubernetes.io/projected/f2407a56-45e8-4257-b271-437daa98e3f4-kube-api-access-2vkdx\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.651699 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.651424 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.651699 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.651446 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.651699 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.651480 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2407a56-45e8-4257-b271-437daa98e3f4-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.651699 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.651579 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2407a56-45e8-4257-b271-437daa98e3f4-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.651699 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.651625 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.753194 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.753154 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2407a56-45e8-4257-b271-437daa98e3f4-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.753194 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.753197 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.753419 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.753321 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.753419 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.753366 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vkdx\" (UniqueName: \"kubernetes.io/projected/f2407a56-45e8-4257-b271-437daa98e3f4-kube-api-access-2vkdx\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.753499 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.753441 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.753499 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.753461 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.753605 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.753511 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2407a56-45e8-4257-b271-437daa98e3f4-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.753605 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.753552 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2407a56-45e8-4257-b271-437daa98e3f4-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.753605 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.753589 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.753760 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.753632 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.753907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.753882 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.753989 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.753961 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.754048 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.754016 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.754048 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.754027 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2407a56-45e8-4257-b271-437daa98e3f4-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.755703 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.755674 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.756183 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.756165 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2407a56-45e8-4257-b271-437daa98e3f4-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.776761 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.776731 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2407a56-45e8-4257-b271-437daa98e3f4-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.776966 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.776936 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vkdx\" (UniqueName: \"kubernetes.io/projected/f2407a56-45e8-4257-b271-437daa98e3f4-kube-api-access-2vkdx\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:24.854911 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:24.854806 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:25.175631 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:18:25.175597 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2407a56_45e8_4257_b271_437daa98e3f4.slice/crio-9e17646bdf11af84e9f116c60593cfe3dfb6a7155cf92273c5eeea7251a80b7d WatchSource:0}: Error finding container 9e17646bdf11af84e9f116c60593cfe3dfb6a7155cf92273c5eeea7251a80b7d: Status 404 returned error can't find the container with id 9e17646bdf11af84e9f116c60593cfe3dfb6a7155cf92273c5eeea7251a80b7d Oct 30 12:18:25.175831 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:25.175746 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z"] Oct 30 12:18:25.178253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:25.178218 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:25.178351 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:25.178301 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:25.178393 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:25.178366 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:25.563211 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:25.563117 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" event={"ID":"f2407a56-45e8-4257-b271-437daa98e3f4","Type":"ContainerStarted","Data":"b83f019a39ed73320794428933bd63fe967227d54f193e0d6afafa8683d82146"} Oct 30 12:18:25.563211 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:25.563183 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" event={"ID":"f2407a56-45e8-4257-b271-437daa98e3f4","Type":"ContainerStarted","Data":"9e17646bdf11af84e9f116c60593cfe3dfb6a7155cf92273c5eeea7251a80b7d"} Oct 30 12:18:25.613577 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:25.613523 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" podStartSLOduration=1.613505155 podStartE2EDuration="1.613505155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:25.611044274 +0000 UTC m=+1456.645182665" watchObservedRunningTime="2026-10-30 12:18:25.613505155 +0000 UTC m=+1456.647643550" Oct 30 12:18:25.854969 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:25.854878 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:26.857493 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:26.857454 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 12:18:26.857907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:26.857516 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" podUID="f2407a56-45e8-4257-b271-437daa98e3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 12:18:27.856233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:27.856190 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 12:18:27.856420 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:27.856265 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" podUID="f2407a56-45e8-4257-b271-437daa98e3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 12:18:28.855488 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:28.855438 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 12:18:28.855993 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:28.855508 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" podUID="f2407a56-45e8-4257-b271-437daa98e3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 12:18:29.857064 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:29.856544 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 12:18:29.857064 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:29.856635 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" podUID="f2407a56-45e8-4257-b271-437daa98e3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 12:18:30.294242 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:30.294177 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:30.510183 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:30.510131 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 30 12:18:30.608150 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:30.607211 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:30.857518 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:30.857479 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 12:18:30.858139 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:30.858103 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" podUID="f2407a56-45e8-4257-b271-437daa98e3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 12:18:30.952291 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:30.952225 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:30.961287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:30.961222 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:31.008460 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:31.008412 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:31.025589 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:31.025522 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:31.083289 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:31.083207 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:31.189342 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:31.189272 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 12:18:31.194213 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:31.194159 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:31.348169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:31.344152 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:31.348169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:31.344616 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:18:31.370230 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:31.370174 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:31.597107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:31.572378 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:31.597107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:31.572970 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:31.627307 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:31.608791 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:31.687125 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:31.628745 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:18:31.687125 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:31.642191 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:31.940170 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:31.940132 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 12:18:31.940798 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:31.940763 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" podUID="f2407a56-45e8-4257-b271-437daa98e3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 12:18:31.969748 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:31.969704 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:31.970417 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:31.970390 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:18:32.013985 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:32.013425 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:32.030181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:32.030140 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:18:32.039607 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:32.031740 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:32.051366 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:32.051332 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:18:32.227283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:32.222546 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:32.227283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:32.222855 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:18:32.227283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:32.223781 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 12:18:32.227283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:32.224033 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:18:32.423097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:32.408265 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 12:18:32.423097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:32.417679 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:18:32.500149 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:32.484829 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:18:32.577122 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:32.573794 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:18:32.577122 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:32.574624 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:18:32.646227 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:32.643552 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:18:32.646227 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:32.643937 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:18:32.652820 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:32.652169 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:18:32.665451 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:32.662569 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:18:32.857913 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:32.857816 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 12:18:32.857913 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:32.857892 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" podUID="f2407a56-45e8-4257-b271-437daa98e3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 12:18:34.896296 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:34.896247 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:18:34.896755 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:34.896310 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" podUID="f2407a56-45e8-4257-b271-437daa98e3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:35.388544 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:35.388504 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:35.388961 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:35.388938 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:35.403888 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:35.403855 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:18:39.575699 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.575669 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw"] Oct 30 12:18:39.580392 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.580361 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.602665 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.602628 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw"] Oct 30 12:18:39.654427 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.654388 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.654612 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.654444 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.654612 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.654481 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.654612 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.654520 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.654612 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.654546 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.654612 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.654562 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.654612 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.654581 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrq2c\" (UniqueName: \"kubernetes.io/projected/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-kube-api-access-rrq2c\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.655011 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.654982 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.655147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.655127 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.756848 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.756783 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.757050 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.756895 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.757050 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.756970 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.757050 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.756995 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.757050 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.757028 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.757253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.757106 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.757253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.757149 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.757253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.757177 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.757253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.757207 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrq2c\" (UniqueName: \"kubernetes.io/projected/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-kube-api-access-rrq2c\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.757402 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.757330 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.757783 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.757759 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.758323 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.758295 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.758591 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.758568 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.758727 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.758707 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.760319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.760292 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.761546 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.761518 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.780554 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.780518 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.780716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.780663 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrq2c\" (UniqueName: \"kubernetes.io/projected/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-kube-api-access-rrq2c\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:39.905006 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:39.904909 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:40.250841 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.250811 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw"] Oct 30 12:18:40.251314 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:18:40.251284 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01f9f03_1c1c_4b0a_bed3_5920537c7ea3.slice/crio-e73b2d59329d564c4544bb19b1ee46568ba26eb4fef2b995bff8a3189995fd3b WatchSource:0}: Error finding container e73b2d59329d564c4544bb19b1ee46568ba26eb4fef2b995bff8a3189995fd3b: Status 404 returned error can't find the container with id e73b2d59329d564c4544bb19b1ee46568ba26eb4fef2b995bff8a3189995fd3b Oct 30 12:18:40.253735 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.253701 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:40.253796 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.253787 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:40.253832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.253819 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:40.696023 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.695985 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2"] Oct 30 12:18:40.700164 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.700137 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.757092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.753459 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2"] Oct 30 12:18:40.787354 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.787313 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whjwf\" (UniqueName: \"kubernetes.io/projected/43df4a6b-2b51-4575-aff8-530380e15466-kube-api-access-whjwf\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.787552 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.787418 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.787552 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.787504 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.787655 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.787611 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.787708 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.787678 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43df4a6b-2b51-4575-aff8-530380e15466-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.787790 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.787775 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43df4a6b-2b51-4575-aff8-530380e15466-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.787859 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.787833 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.787974 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.787953 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43df4a6b-2b51-4575-aff8-530380e15466-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.788023 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.788012 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.889886 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.889848 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whjwf\" (UniqueName: \"kubernetes.io/projected/43df4a6b-2b51-4575-aff8-530380e15466-kube-api-access-whjwf\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.890091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.889905 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.890091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.889924 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.890091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.889963 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.890091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.890001 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43df4a6b-2b51-4575-aff8-530380e15466-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.890091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.890035 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43df4a6b-2b51-4575-aff8-530380e15466-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.890365 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.890095 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.890365 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.890150 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43df4a6b-2b51-4575-aff8-530380e15466-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.890365 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.890191 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.890503 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.890355 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.890556 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.890536 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.890621 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.890596 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.890682 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.890630 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.891013 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.890988 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43df4a6b-2b51-4575-aff8-530380e15466-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.892325 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.892307 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.892814 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.892794 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43df4a6b-2b51-4575-aff8-530380e15466-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.969959 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.948457 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43df4a6b-2b51-4575-aff8-530380e15466-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:40.971711 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:40.971509 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whjwf\" (UniqueName: \"kubernetes.io/projected/43df4a6b-2b51-4575-aff8-530380e15466-kube-api-access-whjwf\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:41.033794 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:41.032401 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:41.040285 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:41.040249 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" event={"ID":"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3","Type":"ContainerStarted","Data":"8eb50f13ee8ccc2a7795c3f8127c0eaeb7c94e63cb3b6adc4d184d8b1e9bdf59"} Oct 30 12:18:41.040445 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:41.040305 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" event={"ID":"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3","Type":"ContainerStarted","Data":"e73b2d59329d564c4544bb19b1ee46568ba26eb4fef2b995bff8a3189995fd3b"} Oct 30 12:18:41.088386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:41.088330 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" podStartSLOduration=2.08831499 podStartE2EDuration="2.08831499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:41.086827702 +0000 UTC m=+1472.120966099" watchObservedRunningTime="2026-10-30 12:18:41.08831499 +0000 UTC m=+1472.122453386" Oct 30 12:18:41.366210 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:41.366181 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2"] Oct 30 12:18:41.366846 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:18:41.366823 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43df4a6b_2b51_4575_aff8_530380e15466.slice/crio-a5ffe9a7209ac5112ab5841e6d123d325a74490eba078288f33b7d52d511ff1f WatchSource:0}: Error finding container a5ffe9a7209ac5112ab5841e6d123d325a74490eba078288f33b7d52d511ff1f: Status 404 returned error can't find the container with id a5ffe9a7209ac5112ab5841e6d123d325a74490eba078288f33b7d52d511ff1f Oct 30 12:18:41.369453 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:41.369418 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:41.369571 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:41.369502 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:41.369571 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:41.369549 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:41.911095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:41.909505 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:41.928132 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:41.922214 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 12:18:41.928132 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:41.922285 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" podUID="f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 12:18:42.056237 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:42.056197 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" event={"ID":"43df4a6b-2b51-4575-aff8-530380e15466","Type":"ContainerStarted","Data":"4368dfa756e10a9f825b2e1ce7e6633f35c9b48cc70947cde7312dc1c64caa0f"} Oct 30 12:18:42.056407 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:42.056295 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" event={"ID":"43df4a6b-2b51-4575-aff8-530380e15466","Type":"ContainerStarted","Data":"a5ffe9a7209ac5112ab5841e6d123d325a74490eba078288f33b7d52d511ff1f"} Oct 30 12:18:42.100372 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:42.100313 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" podStartSLOduration=2.100298956 podStartE2EDuration="2.100298956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:42.098880145 +0000 UTC m=+1473.133018541" watchObservedRunningTime="2026-10-30 12:18:42.100298956 +0000 UTC m=+1473.134437353" Oct 30 12:18:42.910097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:42.907571 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 12:18:42.910097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:42.907639 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" podUID="f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 12:18:43.049504 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:43.049466 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:43.051725 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:43.051693 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 12:18:43.051946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:43.051917 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" podUID="43df4a6b-2b51-4575-aff8-530380e15466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 12:18:43.914560 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:43.914519 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 12:18:43.914848 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:43.914820 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" podUID="f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 12:18:44.051935 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:44.051468 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 12:18:44.051935 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:44.051534 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" podUID="43df4a6b-2b51-4575-aff8-530380e15466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 12:18:44.929093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:44.928543 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 12:18:44.929093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:44.928629 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" podUID="f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 12:18:45.050095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:45.036118 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 12:18:45.050095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:45.036181 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" podUID="43df4a6b-2b51-4575-aff8-530380e15466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 12:18:45.919094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:45.911650 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 12:18:45.919094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:45.911717 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" podUID="f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 12:18:46.038109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:46.037545 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 12:18:46.038109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:46.037611 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" podUID="43df4a6b-2b51-4575-aff8-530380e15466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 12:18:46.910097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:46.907520 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 12:18:46.910097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:46.907584 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" podUID="f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 12:18:47.042095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.041514 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 12:18:47.042095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.041596 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" podUID="43df4a6b-2b51-4575-aff8-530380e15466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 12:18:47.502700 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.502663 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt"] Oct 30 12:18:47.527281 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.526517 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.545202 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.543587 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt"] Oct 30 12:18:47.554201 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.553728 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-6a7148-openshift-default-dockercfg-49s7b\"" Oct 30 12:18:47.668094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.666984 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.668094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.667088 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.668094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.667201 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.668094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.667236 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea87399a-8203-4239-a086-097a4bb9120e-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.668094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.667269 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.668094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.667386 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea87399a-8203-4239-a086-097a4bb9120e-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.668094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.667437 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.668094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.667484 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea87399a-8203-4239-a086-097a4bb9120e-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.668094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.667514 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjxmn\" (UniqueName: \"kubernetes.io/projected/ea87399a-8203-4239-a086-097a4bb9120e-kube-api-access-jjxmn\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.804156 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.793650 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.832226 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.832175 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea87399a-8203-4239-a086-097a4bb9120e-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.832504 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.832482 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.832758 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.832741 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea87399a-8203-4239-a086-097a4bb9120e-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.832903 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.832879 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.833041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.833028 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea87399a-8203-4239-a086-097a4bb9120e-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.833177 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.833160 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjxmn\" (UniqueName: \"kubernetes.io/projected/ea87399a-8203-4239-a086-097a4bb9120e-kube-api-access-jjxmn\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.833396 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.833325 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.833548 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.833531 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.838085 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.835845 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea87399a-8203-4239-a086-097a4bb9120e-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.838085 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.806880 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.841587 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.841547 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.842515 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.842487 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea87399a-8203-4239-a086-097a4bb9120e-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.842771 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.842746 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.842886 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.842754 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.851101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.846914 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.864824 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.864789 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjxmn\" (UniqueName: \"kubernetes.io/projected/ea87399a-8203-4239-a086-097a4bb9120e-kube-api-access-jjxmn\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:47.865470 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:47.865442 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea87399a-8203-4239-a086-097a4bb9120e-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:48.035274 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:48.035235 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 12:18:48.035577 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:48.035542 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" podUID="43df4a6b-2b51-4575-aff8-530380e15466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 12:18:48.163244 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:48.163213 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:48.476718 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:48.475330 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:48.479379 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:48.478163 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:48.479795 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:48.479743 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:18:48.590350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:48.589603 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:48.590350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:48.589699 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:48.590350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:48.589748 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:48.597128 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:48.596265 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt"] Oct 30 12:18:49.049148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.044056 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 12:18:49.049148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.044128 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" podUID="43df4a6b-2b51-4575-aff8-530380e15466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 12:18:49.068085 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.068030 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz"] Oct 30 12:18:49.078459 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.078426 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.169128 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.169083 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.169681 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.169140 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468d055f-47b8-4481-84cb-80afefd5ace3-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.169681 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.169178 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.169681 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.169250 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468d055f-47b8-4481-84cb-80afefd5ace3-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.169681 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.169347 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.169681 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.169408 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.169681 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.169456 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.169681 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.169540 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468d055f-47b8-4481-84cb-80afefd5ace3-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.169681 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.169571 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68j2w\" (UniqueName: \"kubernetes.io/projected/468d055f-47b8-4481-84cb-80afefd5ace3-kube-api-access-68j2w\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.370191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.319194 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468d055f-47b8-4481-84cb-80afefd5ace3-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.370191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.319252 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68j2w\" (UniqueName: \"kubernetes.io/projected/468d055f-47b8-4481-84cb-80afefd5ace3-kube-api-access-68j2w\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.370191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.319352 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.370191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.319378 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468d055f-47b8-4481-84cb-80afefd5ace3-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.370191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.319410 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.370191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.319468 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468d055f-47b8-4481-84cb-80afefd5ace3-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.370191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.319561 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.370191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.319630 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.370191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.319675 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.370191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.320304 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.370191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.322002 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.370191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.356523 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.370191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.358325 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468d055f-47b8-4481-84cb-80afefd5ace3-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.370191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.369665 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.379967 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.379875 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468d055f-47b8-4481-84cb-80afefd5ace3-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.404578 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.399767 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468d055f-47b8-4481-84cb-80afefd5ace3-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.404578 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.403666 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz"] Oct 30 12:18:49.421205 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.419619 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" event={"ID":"ea87399a-8203-4239-a086-097a4bb9120e","Type":"ContainerStarted","Data":"faecc3dbf16cbbca0b9a82f68f05f67854150721c0835d14cbc0d240e04106a7"} Oct 30 12:18:49.421205 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.419675 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" event={"ID":"ea87399a-8203-4239-a086-097a4bb9120e","Type":"ContainerStarted","Data":"262639102a7d38a4ecfde246480b5d98614c238887a6748e2aae9bc128730340"} Oct 30 12:18:49.522733 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.522578 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" podStartSLOduration=2.52255299 podStartE2EDuration="2.52255299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:49.464909961 +0000 UTC m=+1480.499048357" watchObservedRunningTime="2026-10-30 12:18:49.52255299 +0000 UTC m=+1480.556691390" Oct 30 12:18:49.612099 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.603329 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68j2w\" (UniqueName: \"kubernetes.io/projected/468d055f-47b8-4481-84cb-80afefd5ace3-kube-api-access-68j2w\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.647149 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.646706 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.715002 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.714714 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:49.971522 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.971002 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:18:49.971522 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:49.971090 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" podUID="64cf635f-868a-4021-91aa-f36a314e26c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:18:50.165270 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:50.165234 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:50.167084 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:50.167037 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 12:18:50.167269 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:50.167113 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" podUID="ea87399a-8203-4239-a086-097a4bb9120e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 12:18:50.240667 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:50.240635 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz"] Oct 30 12:18:50.242701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:50.242669 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:50.242795 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:50.242734 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:50.242795 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:50.242764 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:18:50.457144 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:50.452975 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:50.457144 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:50.454856 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:50.459932 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:50.457789 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:18:50.487732 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:50.487633 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" event={"ID":"468d055f-47b8-4481-84cb-80afefd5ace3","Type":"ContainerStarted","Data":"2d29b02e1ef908a8340c3fc375111b40b7b773ba81f94ba3e5548413854d03e1"} Oct 30 12:18:50.487732 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:50.487686 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" event={"ID":"468d055f-47b8-4481-84cb-80afefd5ace3","Type":"ContainerStarted","Data":"3f4bcdf341890ede44e78a8d380e46699b181f3a1db7fa616d0e31d4db02e0d2"} Oct 30 12:18:50.568652 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:50.568374 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" podStartSLOduration=1.5683516800000001 podStartE2EDuration="1.56835168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:50.56362213 +0000 UTC m=+1481.597760529" watchObservedRunningTime="2026-10-30 12:18:50.56835168 +0000 UTC m=+1481.602490087" Oct 30 12:18:50.749318 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:50.749224 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:51.167576 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:51.167535 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 12:18:51.167734 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:51.167600 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" podUID="ea87399a-8203-4239-a086-097a4bb9120e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 12:18:51.742096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:51.738226 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 12:18:51.742096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:51.738300 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" podUID="468d055f-47b8-4481-84cb-80afefd5ace3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 12:18:52.167693 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:52.166612 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 12:18:52.167693 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:52.166703 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" podUID="ea87399a-8203-4239-a086-097a4bb9120e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 12:18:52.726096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:52.725595 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 12:18:52.726096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:52.725665 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" podUID="468d055f-47b8-4481-84cb-80afefd5ace3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 12:18:53.165610 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:53.165573 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 12:18:53.166012 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:53.165627 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" podUID="ea87399a-8203-4239-a086-097a4bb9120e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 12:18:53.602780 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:18:53.602693 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:18:53.732095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:53.720661 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 12:18:53.732095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:53.720741 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" podUID="468d055f-47b8-4481-84cb-80afefd5ace3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 12:18:54.165874 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:54.165822 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 12:18:54.166398 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:54.165898 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" podUID="ea87399a-8203-4239-a086-097a4bb9120e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 12:18:54.719255 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:54.718550 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 12:18:54.719255 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:54.718635 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" podUID="468d055f-47b8-4481-84cb-80afefd5ace3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 12:18:55.165517 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:55.165467 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 12:18:55.165695 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:55.165577 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" podUID="ea87399a-8203-4239-a086-097a4bb9120e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 12:18:55.733094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:55.718685 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 12:18:55.733094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:55.718750 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" podUID="468d055f-47b8-4481-84cb-80afefd5ace3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 12:18:56.165576 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:56.165539 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 12:18:56.165767 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:56.165602 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" podUID="ea87399a-8203-4239-a086-097a4bb9120e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 12:18:56.717411 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:56.717364 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 12:18:56.717684 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:56.717457 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" podUID="468d055f-47b8-4481-84cb-80afefd5ace3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 12:18:57.169894 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:57.169863 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:57.170417 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:57.170347 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:57.171174 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:57.171148 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 12:18:58.208372 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:58.208340 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:58.210242 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:58.210211 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:18:58.212867 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:18:58.212830 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:19:03.410096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:03.409175 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:03.513096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:03.500249 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 12:19:03.602097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:03.601166 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:03.621717 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:03.602444 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 12:19:03.621717 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:03.617175 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:03.665912 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:03.665800 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 12:19:03.666632 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:03.666236 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:03.679149 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:03.666268 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 12:19:03.739112 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:03.689292 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:04.428408 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:04.428335 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 30 12:19:04.429859 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:04.429830 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:19:04.511623 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:04.511557 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:04.512287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:04.512262 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:19:04.555174 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:04.555132 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn"] Oct 30 12:19:04.703313 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:04.703197 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 12:19:04.739986 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:04.690746 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 12:19:04.897244 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:04.718168 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 12:19:04.898185 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:04.786722 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 12:19:04.899576 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:04.786752 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 12:19:04.900051 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:04.799344 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 12:19:05.144179 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.136548 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:19:05.152096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.148157 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" podUID="64cf635f-868a-4021-91aa-f36a314e26c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:05.327095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.319499 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:19:05.334520 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.334472 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.393902 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.393848 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.394263 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.394188 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.394430 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.394410 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4brs\" (UniqueName: \"kubernetes.io/projected/b3984ddb-e0d7-407d-93db-510c1bc61f0e-kube-api-access-l4brs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.394656 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.394639 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.394948 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.394927 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.395106 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.395090 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.395335 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.395316 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.396104 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.396057 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.396331 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.396314 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.454314 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.454276 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:05.529100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.497728 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.538097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.529510 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.547110 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.538397 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.551137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.547403 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4brs\" (UniqueName: \"kubernetes.io/projected/b3984ddb-e0d7-407d-93db-510c1bc61f0e-kube-api-access-l4brs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.658393 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.583429 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.658660 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.517589 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:05.659606 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.618842 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.679126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.675158 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.679126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.675397 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.679126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.675438 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.679126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.675533 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.679126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.675575 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.679126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.676123 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.699138 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.686243 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:05.761037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.760998 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:19:05.761729 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.761707 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:19:05.762811 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.762793 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:19:05.839524 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.839494 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:19:05.862244 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.862206 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn"] Oct 30 12:19:05.932354 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.932259 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:19:05.998119 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.761317 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 12:19:06.183367 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.826924 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4brs\" (UniqueName: \"kubernetes.io/projected/b3984ddb-e0d7-407d-93db-510c1bc61f0e-kube-api-access-l4brs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:06.183606 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.889321 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:06.313203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.889393 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:06.356322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.920527 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 12:19:06.356322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.920570 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:06.356322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.920591 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 12:19:06.356322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.920609 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 12:19:06.356322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.932626 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:06.356322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.839570 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:06.356322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:05.935829 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:06.356322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:06.225201 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:06.356322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:06.282383 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:19:06.490271 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:06.490153 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:06.491268 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:06.491239 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:19:06.498098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:06.491435 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" podUID="f2407a56-45e8-4257-b271-437daa98e3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:06.553717 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:06.553511 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:06.586908 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:06.586802 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:19:06.667628 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:06.660858 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8"] Oct 30 12:19:07.003979 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.003937 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:07.063101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.055446 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8"] Oct 30 12:19:07.063101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.055550 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:19:07.063101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.055591 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:19:07.063101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.055628 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:19:07.063101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.055661 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:19:07.063101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.055695 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:19:07.063101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.056671 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.426090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.424739 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.426090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.425696 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shzlx\" (UniqueName: \"kubernetes.io/projected/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-kube-api-access-shzlx\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.426090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.425843 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.426090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.425950 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.431406 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.431368 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.460447 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.452428 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.460447 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.452589 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.460447 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.452697 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.460447 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.452740 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.509157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.494860 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:19:07.509157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.494909 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn"] Oct 30 12:19:07.534667 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.534622 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:07.534929 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.534906 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:07.535446 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.535421 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:07.554923 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.554890 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.555242 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.555220 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.555376 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.555360 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.555561 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.555547 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.555827 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.555809 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shzlx\" (UniqueName: \"kubernetes.io/projected/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-kube-api-access-shzlx\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.556085 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.556056 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.556224 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.556183 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.556305 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.556210 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.556439 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.556424 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.556538 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.556525 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.558402 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.558378 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.558747 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.555865 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.559287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.558900 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.559287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.559241 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.559446 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.559288 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.561005 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.560983 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.582282 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.582246 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.582515 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.582377 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shzlx\" (UniqueName: \"kubernetes.io/projected/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-kube-api-access-shzlx\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:07.789453 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:07.789359 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:08.007287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:08.007207 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" event={"ID":"b3984ddb-e0d7-407d-93db-510c1bc61f0e","Type":"ContainerStarted","Data":"ba0fe44273da8c39d4c39927c0c375bf8f7c77c28befb8ae81823643fc7bebe4"} Oct 30 12:19:08.007636 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:08.007591 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" event={"ID":"b3984ddb-e0d7-407d-93db-510c1bc61f0e","Type":"ContainerStarted","Data":"183bc8b05205ee81ea87bc46d4daa3d1f899dba64186428b4d2e75515eecc375"} Oct 30 12:19:08.055136 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:08.055025 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" podStartSLOduration=4.055010588 podStartE2EDuration="4.055010588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:08.051435257 +0000 UTC m=+1499.085573678" watchObservedRunningTime="2026-10-30 12:19:08.055010588 +0000 UTC m=+1499.089148983" Oct 30 12:19:08.218712 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:08.218678 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8"] Oct 30 12:19:08.219106 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:19:08.219044 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c16fb3_1a1a_4e66_a390_7004db9a2bec.slice/crio-f361a399152e1c6beee03c584fcdd0a8db90d727747c8669d6cf0bd218150331 WatchSource:0}: Error finding container f361a399152e1c6beee03c584fcdd0a8db90d727747c8669d6cf0bd218150331: Status 404 returned error can't find the container with id f361a399152e1c6beee03c584fcdd0a8db90d727747c8669d6cf0bd218150331 Oct 30 12:19:08.221691 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:08.221653 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:08.221788 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:08.221736 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:08.221858 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:08.221783 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:08.569577 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:08.569534 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:08.570797 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:08.570769 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 12:19:08.570916 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:08.570826 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" podUID="b3984ddb-e0d7-407d-93db-510c1bc61f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 12:19:09.037264 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:09.037162 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" event={"ID":"a0c16fb3-1a1a-4e66-a390-7004db9a2bec","Type":"ContainerStarted","Data":"e0abe153c69d7af6abe51e908be8fa6a5b6889117425ebc4fc4a58f24e18373d"} Oct 30 12:19:09.037264 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:09.037226 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" event={"ID":"a0c16fb3-1a1a-4e66-a390-7004db9a2bec","Type":"ContainerStarted","Data":"f361a399152e1c6beee03c584fcdd0a8db90d727747c8669d6cf0bd218150331"} Oct 30 12:19:09.080759 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:09.080708 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" podStartSLOduration=3.080693135 podStartE2EDuration="3.080693135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:09.080206986 +0000 UTC m=+1500.114345383" watchObservedRunningTime="2026-10-30 12:19:09.080693135 +0000 UTC m=+1500.114831530" Oct 30 12:19:09.570082 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:09.570025 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 12:19:09.570499 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:09.570120 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" podUID="b3984ddb-e0d7-407d-93db-510c1bc61f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 12:19:09.793092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:09.790418 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:09.793092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:09.792386 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 12:19:09.793092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:09.792448 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" podUID="a0c16fb3-1a1a-4e66-a390-7004db9a2bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 12:19:09.831157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:09.830352 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf"] Oct 30 12:19:09.871259 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:09.870973 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf"] Oct 30 12:19:09.871259 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:09.871176 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf" Oct 30 12:19:09.885542 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:09.885512 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:19:09.885724 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:09.885650 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:19:09.959522 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:09.959330 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd0f4fd-f5b8-421f-bbea-f848517e8392-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00c25542-k4dcf\" (UID: \"0dd0f4fd-f5b8-421f-bbea-f848517e8392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf" Oct 30 12:19:09.959522 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:09.959480 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd0f4fd-f5b8-421f-bbea-f848517e8392-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00c25542-k4dcf\" (UID: \"0dd0f4fd-f5b8-421f-bbea-f848517e8392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf" Oct 30 12:19:09.959765 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:09.959700 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psx26\" (UniqueName: \"kubernetes.io/projected/0dd0f4fd-f5b8-421f-bbea-f848517e8392-kube-api-access-psx26\") pod \"maas-api-revoke-sub-00c25542-k4dcf\" (UID: \"0dd0f4fd-f5b8-421f-bbea-f848517e8392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf" Oct 30 12:19:10.061355 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:10.060789 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd0f4fd-f5b8-421f-bbea-f848517e8392-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00c25542-k4dcf\" (UID: \"0dd0f4fd-f5b8-421f-bbea-f848517e8392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf" Oct 30 12:19:10.061355 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:10.060891 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd0f4fd-f5b8-421f-bbea-f848517e8392-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00c25542-k4dcf\" (UID: \"0dd0f4fd-f5b8-421f-bbea-f848517e8392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf" Oct 30 12:19:10.061355 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:10.061015 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psx26\" (UniqueName: \"kubernetes.io/projected/0dd0f4fd-f5b8-421f-bbea-f848517e8392-kube-api-access-psx26\") pod \"maas-api-revoke-sub-00c25542-k4dcf\" (UID: \"0dd0f4fd-f5b8-421f-bbea-f848517e8392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf" Oct 30 12:19:10.061733 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:10.061706 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd0f4fd-f5b8-421f-bbea-f848517e8392-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00c25542-k4dcf\" (UID: \"0dd0f4fd-f5b8-421f-bbea-f848517e8392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf" Oct 30 12:19:10.065294 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:10.065272 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd0f4fd-f5b8-421f-bbea-f848517e8392-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00c25542-k4dcf\" (UID: \"0dd0f4fd-f5b8-421f-bbea-f848517e8392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf" Oct 30 12:19:10.084107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:10.084009 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psx26\" (UniqueName: \"kubernetes.io/projected/0dd0f4fd-f5b8-421f-bbea-f848517e8392-kube-api-access-psx26\") pod \"maas-api-revoke-sub-00c25542-k4dcf\" (UID: \"0dd0f4fd-f5b8-421f-bbea-f848517e8392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf" Oct 30 12:19:10.191490 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:10.191452 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf" Oct 30 12:19:10.569870 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:10.569832 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 12:19:10.570109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:10.569889 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" podUID="b3984ddb-e0d7-407d-93db-510c1bc61f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 12:19:10.599975 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:19:10.599937 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd0f4fd_f5b8_421f_bbea_f848517e8392.slice/crio-2fdc6c5de6ea3914cfc7a84b21cd54df8d9497e930375cf52db1d27c337e4bf3 WatchSource:0}: Error finding container 2fdc6c5de6ea3914cfc7a84b21cd54df8d9497e930375cf52db1d27c337e4bf3: Status 404 returned error can't find the container with id 2fdc6c5de6ea3914cfc7a84b21cd54df8d9497e930375cf52db1d27c337e4bf3 Oct 30 12:19:10.600951 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:10.600921 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf"] Oct 30 12:19:10.790970 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:10.790930 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 12:19:10.791179 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:10.790995 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" podUID="a0c16fb3-1a1a-4e66-a390-7004db9a2bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 12:19:11.094515 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:11.094477 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf" event={"ID":"0dd0f4fd-f5b8-421f-bbea-f848517e8392","Type":"ContainerStarted","Data":"2fdc6c5de6ea3914cfc7a84b21cd54df8d9497e930375cf52db1d27c337e4bf3"} Oct 30 12:19:11.569517 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:11.569484 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 12:19:11.569715 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:11.569535 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" podUID="b3984ddb-e0d7-407d-93db-510c1bc61f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 12:19:11.794129 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:11.790563 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 12:19:11.794129 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:11.790630 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" podUID="a0c16fb3-1a1a-4e66-a390-7004db9a2bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 12:19:12.094047 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:12.094011 4468 generic.go:356] "Generic (PLEG): container finished" podID="0dd0f4fd-f5b8-421f-bbea-f848517e8392" containerID="87712a02ccb963f53d4cd38735ddea858f283e1c2fae805cd4a0237079f66ccb" exitCode=0 Oct 30 12:19:12.094370 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:12.094116 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf" event={"ID":"0dd0f4fd-f5b8-421f-bbea-f848517e8392","Type":"ContainerDied","Data":"87712a02ccb963f53d4cd38735ddea858f283e1c2fae805cd4a0237079f66ccb"} Oct 30 12:19:12.569598 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:12.569555 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 12:19:12.569788 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:12.569622 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" podUID="b3984ddb-e0d7-407d-93db-510c1bc61f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 12:19:12.790192 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:12.790154 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 12:19:12.790354 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:12.790215 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" podUID="a0c16fb3-1a1a-4e66-a390-7004db9a2bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 12:19:13.397647 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:13.397618 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf" Oct 30 12:19:13.472234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:13.472203 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dd0f4fd-f5b8-421f-bbea-f848517e8392-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd0f4fd-f5b8-421f-bbea-f848517e8392-maas-api-xks-ca\") pod \"0dd0f4fd-f5b8-421f-bbea-f848517e8392\" (UID: \"0dd0f4fd-f5b8-421f-bbea-f848517e8392\") " Oct 30 12:19:13.472407 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:13.472273 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd0f4fd-f5b8-421f-bbea-f848517e8392-kube-api-access-psx26\" (UniqueName: \"kubernetes.io/projected/0dd0f4fd-f5b8-421f-bbea-f848517e8392-kube-api-access-psx26\") pod \"0dd0f4fd-f5b8-421f-bbea-f848517e8392\" (UID: \"0dd0f4fd-f5b8-421f-bbea-f848517e8392\") " Oct 30 12:19:13.472407 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:13.472389 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd0f4fd-f5b8-421f-bbea-f848517e8392-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd0f4fd-f5b8-421f-bbea-f848517e8392-maas-api-openshift-ca\") pod \"0dd0f4fd-f5b8-421f-bbea-f848517e8392\" (UID: \"0dd0f4fd-f5b8-421f-bbea-f848517e8392\") " Oct 30 12:19:13.473135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:13.473112 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd0f4fd-f5b8-421f-bbea-f848517e8392-maas-api-openshift-ca" pod "0dd0f4fd-f5b8-421f-bbea-f848517e8392" (UID: "0dd0f4fd-f5b8-421f-bbea-f848517e8392"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:19:13.475729 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:13.475702 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd0f4fd-f5b8-421f-bbea-f848517e8392-maas-api-xks-ca" pod "0dd0f4fd-f5b8-421f-bbea-f848517e8392" (UID: "0dd0f4fd-f5b8-421f-bbea-f848517e8392"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:19:13.476246 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:13.476215 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd0f4fd-f5b8-421f-bbea-f848517e8392-kube-api-access-psx26" pod "0dd0f4fd-f5b8-421f-bbea-f848517e8392" (UID: "0dd0f4fd-f5b8-421f-bbea-f848517e8392"). InnerVolumeSpecName "kube-api-access-psx26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:19:13.570221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:13.570182 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 12:19:13.570431 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:13.570249 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" podUID="b3984ddb-e0d7-407d-93db-510c1bc61f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 12:19:13.574166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:13.574135 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd0f4fd-f5b8-421f-bbea-f848517e8392-maas-api-xks-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:19:13.574166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:13.574161 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psx26\" (UniqueName: \"kubernetes.io/projected/0dd0f4fd-f5b8-421f-bbea-f848517e8392-kube-api-access-psx26\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:19:13.574166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:13.574171 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd0f4fd-f5b8-421f-bbea-f848517e8392-maas-api-openshift-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:19:13.823098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:13.821453 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 12:19:13.823098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:13.821519 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" podUID="a0c16fb3-1a1a-4e66-a390-7004db9a2bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 12:19:14.159350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:14.152505 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf" event={"ID":"0dd0f4fd-f5b8-421f-bbea-f848517e8392","Type":"ContainerDied","Data":"2fdc6c5de6ea3914cfc7a84b21cd54df8d9497e930375cf52db1d27c337e4bf3"} Oct 30 12:19:14.159350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:14.152589 4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fdc6c5de6ea3914cfc7a84b21cd54df8d9497e930375cf52db1d27c337e4bf3" Oct 30 12:19:14.159350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:14.152679 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf" Oct 30 12:19:14.569569 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:14.569483 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 12:19:14.569569 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:14.569553 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" podUID="b3984ddb-e0d7-407d-93db-510c1bc61f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 12:19:14.790199 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:14.790155 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 12:19:14.790411 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:14.790226 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" podUID="a0c16fb3-1a1a-4e66-a390-7004db9a2bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 12:19:15.784938 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:15.784878 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:15.786574 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:15.786526 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:15.787938 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:15.787914 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:19:15.790343 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:15.790312 4468 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 12:19:15.790481 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:15.790384 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" podUID="a0c16fb3-1a1a-4e66-a390-7004db9a2bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 12:19:16.795187 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:16.795154 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:16.795579 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:16.795557 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:16.796333 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:16.796317 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:19:23.484095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:23.458988 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g"] Oct 30 12:19:23.484095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:23.460017 4468 cpu_manager.go:414] "containerMap: removing container" podUID="0dd0f4fd-f5b8-421f-bbea-f848517e8392" containerName="revoke-keys" Oct 30 12:19:23.484095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:23.460041 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd0f4fd-f5b8-421f-bbea-f848517e8392" containerName="revoke-keys" Oct 30 12:19:23.484095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:23.460229 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd0f4fd-f5b8-421f-bbea-f848517e8392" containerName="revoke-keys" Oct 30 12:19:23.826193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:23.808428 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" Oct 30 12:19:23.826193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:23.810085 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g"] Oct 30 12:19:23.858213 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:23.851192 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-060f41f2\"" Oct 30 12:19:23.892094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:23.887062 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-060f41f2-dockercfg-qnt6l\"" Oct 30 12:19:23.916476 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:23.915615 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6337aeed-f473-40d3-89eb-ac6f6e8bd7a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g\" (UID: \"6337aeed-f473-40d3-89eb-ac6f6e8bd7a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" Oct 30 12:19:23.916476 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:23.915731 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwhb2\" (UniqueName: \"kubernetes.io/projected/6337aeed-f473-40d3-89eb-ac6f6e8bd7a1-kube-api-access-fwhb2\") pod \"payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g\" (UID: \"6337aeed-f473-40d3-89eb-ac6f6e8bd7a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" Oct 30 12:19:24.017736 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:24.017679 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwhb2\" (UniqueName: \"kubernetes.io/projected/6337aeed-f473-40d3-89eb-ac6f6e8bd7a1-kube-api-access-fwhb2\") pod \"payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g\" (UID: \"6337aeed-f473-40d3-89eb-ac6f6e8bd7a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" Oct 30 12:19:24.018387 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:24.018362 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6337aeed-f473-40d3-89eb-ac6f6e8bd7a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g\" (UID: \"6337aeed-f473-40d3-89eb-ac6f6e8bd7a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" Oct 30 12:19:24.019984 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:24.019961 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6337aeed-f473-40d3-89eb-ac6f6e8bd7a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g\" (UID: \"6337aeed-f473-40d3-89eb-ac6f6e8bd7a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" Oct 30 12:19:24.055164 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:24.055127 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwhb2\" (UniqueName: \"kubernetes.io/projected/6337aeed-f473-40d3-89eb-ac6f6e8bd7a1-kube-api-access-fwhb2\") pod \"payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g\" (UID: \"6337aeed-f473-40d3-89eb-ac6f6e8bd7a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" Oct 30 12:19:24.059215 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:24.059166 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:24.227339 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:19:24.227292 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:19:24.275549 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:24.275513 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" Oct 30 12:19:24.347226 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:24.347179 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:24.348199 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:24.348145 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 12:19:24.401116 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:24.399218 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:24.502420 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:24.502301 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 30 12:19:24.618099 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:24.615164 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 12:19:25.037155 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:25.036350 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 12:19:25.045094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:25.043181 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:25.067103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:25.065309 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:25.067103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:25.065791 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:19:25.376764 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:25.376587 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:25.410101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:25.402850 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 12:19:25.445518 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:25.445464 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:25.512865 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:25.512814 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 30 12:19:25.688159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:25.641138 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:19:25.688159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:25.641246 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" podUID="56010c34-d070-4709-8ec7-6c9a23178226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:25.688159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:25.641621 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:26.144129 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:26.143535 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:26.177096 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:19:26.151866 4468 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.062s" Oct 30 12:19:26.406131 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:26.391185 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g"] Oct 30 12:19:26.406131 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:26.391328 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:19:26.406131 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:26.391367 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:19:26.406131 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:26.391410 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:19:26.406131 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:26.391501 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:19:26.406131 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:26.391534 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:19:26.406131 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:26.391567 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:19:26.406131 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:26.169527 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 12:19:26.406131 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:26.187674 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:26.435783 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:26.407719 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:19:26.569397 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:26.569291 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 12:19:26.570294 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:26.570260 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:26.598241 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:26.598190 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:26.647462 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:26.647412 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 12:19:26.791236 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:26.791185 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 12:19:26.801534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:26.801487 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 30 12:19:27.276217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:27.276167 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Oct 30 12:19:27.542233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:27.313308 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:27.542233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:27.489477 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:27.649205 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:27.649162 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:27.650502 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:27.650194 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:27.663102 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:27.655890 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:27.663102 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:27.656629 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 12:19:27.880835 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:27.880787 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:28.030190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:28.017090 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:19:28.030190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:28.017148 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:19:28.030190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:28.017166 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" event={"ID":"6337aeed-f473-40d3-89eb-ac6f6e8bd7a1","Type":"ContainerStarted","Data":"6c070e7abfb7841ee068aec183102bd1df4e8317fc0f2855f97ee2362f2a438b"} Oct 30 12:19:28.316540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:28.316491 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:28.476343 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:28.476307 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:19:28.607151 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:28.599579 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:19:28.607151 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:28.599643 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" podUID="ea87399a-8203-4239-a086-097a4bb9120e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:28.654332 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:28.653952 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 12:19:28.668269 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:28.668213 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:28.677100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:28.668513 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:28.899564 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:28.828527 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 12:19:28.921838 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:28.921786 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:29.379095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:29.345152 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 12:19:29.547148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:29.522187 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:29.695149 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:29.686159 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:29.722092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:29.706902 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:19:29.723223 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:29.723191 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:19:29.753312 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:29.753278 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:19:29.759215 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:29.753573 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" event={"ID":"6337aeed-f473-40d3-89eb-ac6f6e8bd7a1","Type":"ContainerStarted","Data":"661849857db2cb462142d2710a40e4a7ff31bcab3cf8a6a49c06450384e8887c"} Oct 30 12:19:29.852680 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:29.852618 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" podStartSLOduration=6.852596565 podStartE2EDuration="6.852596565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:29.584345628 +0000 UTC m=+1520.618484030" watchObservedRunningTime="2026-10-30 12:19:29.852596565 +0000 UTC m=+1520.886734961" Oct 30 12:19:30.020682 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:30.020592 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:19:30.035129 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:30.034777 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:19:30.525832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:30.525791 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" Oct 30 12:19:35.444227 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:35.444186 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk"] Oct 30 12:19:35.448610 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:35.448586 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" Oct 30 12:19:35.489178 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:35.485329 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-060f41f2-serving-cert\"" Oct 30 12:19:35.489728 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:35.489359 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk"] Oct 30 12:19:35.527621 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:35.527584 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg8d7\" (UniqueName: \"kubernetes.io/projected/5593fe4c-beb7-4fa5-92b1-e023a25e186c-kube-api-access-sg8d7\") pod \"maas-api-e2e-mt-060f41f2-744745b898-qqfwk\" (UID: \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" Oct 30 12:19:35.527803 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:35.527658 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5593fe4c-beb7-4fa5-92b1-e023a25e186c-metrics-certs\") pod \"maas-api-e2e-mt-060f41f2-744745b898-qqfwk\" (UID: \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" Oct 30 12:19:35.527847 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:35.527809 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5593fe4c-beb7-4fa5-92b1-e023a25e186c-openshift-service-ca\") pod \"maas-api-e2e-mt-060f41f2-744745b898-qqfwk\" (UID: \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" Oct 30 12:19:35.527890 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:35.527877 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5593fe4c-beb7-4fa5-92b1-e023a25e186c-maas-api-tls\") pod \"maas-api-e2e-mt-060f41f2-744745b898-qqfwk\" (UID: \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" Oct 30 12:19:35.654100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:35.645089 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg8d7\" (UniqueName: \"kubernetes.io/projected/5593fe4c-beb7-4fa5-92b1-e023a25e186c-kube-api-access-sg8d7\") pod \"maas-api-e2e-mt-060f41f2-744745b898-qqfwk\" (UID: \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" Oct 30 12:19:35.654847 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:35.654817 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5593fe4c-beb7-4fa5-92b1-e023a25e186c-metrics-certs\") pod \"maas-api-e2e-mt-060f41f2-744745b898-qqfwk\" (UID: \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" Oct 30 12:19:35.669298 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:35.669261 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5593fe4c-beb7-4fa5-92b1-e023a25e186c-openshift-service-ca\") pod \"maas-api-e2e-mt-060f41f2-744745b898-qqfwk\" (UID: \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" Oct 30 12:19:35.670423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:35.670398 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5593fe4c-beb7-4fa5-92b1-e023a25e186c-maas-api-tls\") pod \"maas-api-e2e-mt-060f41f2-744745b898-qqfwk\" (UID: \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" Oct 30 12:19:35.670904 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:35.670294 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5593fe4c-beb7-4fa5-92b1-e023a25e186c-openshift-service-ca\") pod \"maas-api-e2e-mt-060f41f2-744745b898-qqfwk\" (UID: \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" Oct 30 12:19:35.671062 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:35.663296 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5593fe4c-beb7-4fa5-92b1-e023a25e186c-metrics-certs\") pod \"maas-api-e2e-mt-060f41f2-744745b898-qqfwk\" (UID: \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" Oct 30 12:19:35.671182 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:19:35.670666 4468 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-serving-cert: secret "maas-api-e2e-mt-060f41f2-serving-cert" not found Oct 30 12:19:35.671341 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:19:35.671325 4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5593fe4c-beb7-4fa5-92b1-e023a25e186c-maas-api-tls podName:5593fe4c-beb7-4fa5-92b1-e023a25e186c nodeName:}" failed. No retries permitted until 2026-10-30 12:19:36.171299154 +0000 UTC m=+1527.205437527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5593fe4c-beb7-4fa5-92b1-e023a25e186c-maas-api-tls") pod "maas-api-e2e-mt-060f41f2-744745b898-qqfwk" (UID: "5593fe4c-beb7-4fa5-92b1-e023a25e186c") : secret "maas-api-e2e-mt-060f41f2-serving-cert" not found Oct 30 12:19:35.687846 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:35.687811 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg8d7\" (UniqueName: \"kubernetes.io/projected/5593fe4c-beb7-4fa5-92b1-e023a25e186c-kube-api-access-sg8d7\") pod \"maas-api-e2e-mt-060f41f2-744745b898-qqfwk\" (UID: \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" Oct 30 12:19:36.178664 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:36.178619 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5593fe4c-beb7-4fa5-92b1-e023a25e186c-maas-api-tls\") pod \"maas-api-e2e-mt-060f41f2-744745b898-qqfwk\" (UID: \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" Oct 30 12:19:36.181190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:36.181157 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5593fe4c-beb7-4fa5-92b1-e023a25e186c-maas-api-tls\") pod \"maas-api-e2e-mt-060f41f2-744745b898-qqfwk\" (UID: \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" Oct 30 12:19:36.365972 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:36.365935 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" Oct 30 12:19:36.755829 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:36.755706 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk"] Oct 30 12:19:36.771908 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:36.771881 4468 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:19:37.574652 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:37.574552 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" event={"ID":"5593fe4c-beb7-4fa5-92b1-e023a25e186c","Type":"ContainerStarted","Data":"1c5b1fc7289db493cbd8b137a37631dec8675a44ac983ba00be4e78069646e3d"} Oct 30 12:19:37.574652 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:37.574616 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" event={"ID":"5593fe4c-beb7-4fa5-92b1-e023a25e186c","Type":"ContainerStarted","Data":"bfbce2e21f8ad70fe9f594e3b91bfa70502b7659e38b5d51a1925442fef1ee77"} Oct 30 12:19:37.574861 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:37.574778 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" Oct 30 12:19:37.627899 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:37.627839 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" podStartSLOduration=2.211392256 podStartE2EDuration="2.627824586s" totalImagesPullingTime="416.43233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:37.625383953 +0000 UTC m=+1528.659522351" watchObservedRunningTime="2026-10-30 12:19:37.627824586 +0000 UTC m=+1528.661962981" Oct 30 12:19:41.537480 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.537447 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" Oct 30 12:19:41.626792 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.626749 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq"] Oct 30 12:19:41.631577 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.631541 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.653363 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.653327 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq"] Oct 30 12:19:41.682916 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.682873 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.683135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.683113 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.683202 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.683152 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.683347 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.683316 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.683397 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.683381 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.683519 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.683493 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.683602 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.683581 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.683798 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.683774 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9txl6\" (UniqueName: \"kubernetes.io/projected/4ad41b97-a567-4fc6-8d54-842e1547bfd7-kube-api-access-9txl6\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.683923 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.683860 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.785216 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.785180 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.785416 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.785250 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.785416 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.785356 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.785416 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.785380 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.785603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.785429 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.785603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.785464 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.785603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.785494 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.785603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.785557 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.785603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.785577 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9txl6\" (UniqueName: \"kubernetes.io/projected/4ad41b97-a567-4fc6-8d54-842e1547bfd7-kube-api-access-9txl6\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.785888 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.785693 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.785888 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.785727 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.786370 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.786343 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.786670 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.786643 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.786844 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.786820 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.790319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.788923 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.790319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.789175 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.808909 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.808862 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.809051 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.808936 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9txl6\" (UniqueName: \"kubernetes.io/projected/4ad41b97-a567-4fc6-8d54-842e1547bfd7-kube-api-access-9txl6\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpkqq\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:41.946881 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:41.946841 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:42.392125 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:42.390099 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq"] Oct 30 12:19:42.392125 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:19:42.390717 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad41b97_a567_4fc6_8d54_842e1547bfd7.slice/crio-cf5ce32df7fbce4a1def2ee207c9f96d145aa95ccf3b8429e8109121cc0fedf1 WatchSource:0}: Error finding container cf5ce32df7fbce4a1def2ee207c9f96d145aa95ccf3b8429e8109121cc0fedf1: Status 404 returned error can't find the container with id cf5ce32df7fbce4a1def2ee207c9f96d145aa95ccf3b8429e8109121cc0fedf1 Oct 30 12:19:42.397515 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:42.396225 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:42.397515 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:42.396320 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:42.397515 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:42.396364 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:42.628931 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:42.628897 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" event={"ID":"4ad41b97-a567-4fc6-8d54-842e1547bfd7","Type":"ContainerStarted","Data":"cf5ce32df7fbce4a1def2ee207c9f96d145aa95ccf3b8429e8109121cc0fedf1"} Oct 30 12:19:43.659105 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:43.656348 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" Oct 30 12:19:43.661480 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:43.660814 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" event={"ID":"4ad41b97-a567-4fc6-8d54-842e1547bfd7","Type":"ContainerStarted","Data":"b5bbd24cb0d88f0180ed98c526480cf8741bdd47be880e4a66db04f569ba4f7c"} Oct 30 12:19:43.757175 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:43.757110 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" podStartSLOduration=2.7570865209999997 podStartE2EDuration="2.757086521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:43.752469484 +0000 UTC m=+1534.786607895" watchObservedRunningTime="2026-10-30 12:19:43.757086521 +0000 UTC m=+1534.791224918" Oct 30 12:19:43.948046 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:43.947957 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:43.949932 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:43.949905 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 12:19:43.950082 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:43.949954 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" podUID="4ad41b97-a567-4fc6-8d54-842e1547bfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 12:19:44.952015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:44.951514 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 12:19:44.952015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:44.951594 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" podUID="4ad41b97-a567-4fc6-8d54-842e1547bfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 12:19:45.947461 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:45.947422 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 12:19:45.947649 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:45.947484 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" podUID="4ad41b97-a567-4fc6-8d54-842e1547bfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 12:19:46.964123 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:46.955609 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 12:19:46.964123 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:46.955693 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" podUID="4ad41b97-a567-4fc6-8d54-842e1547bfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 12:19:47.853349 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:47.853290 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:47.976061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:47.976018 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 12:19:47.983109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:47.976686 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" podUID="4ad41b97-a567-4fc6-8d54-842e1547bfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 12:19:48.057401 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:48.057354 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:48.438095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:48.437165 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 12:19:48.493408 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:48.493357 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:48.502311 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:48.502260 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:48.581104 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:48.575176 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:48.699164 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:48.697171 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:48.799094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:48.797219 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:48.858762 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:48.858711 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 30 12:19:48.860945 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:48.860914 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:19:49.104621 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.104520 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 12:19:49.105258 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.105220 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" podUID="4ad41b97-a567-4fc6-8d54-842e1547bfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 12:19:49.107471 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.107437 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 12:19:49.411441 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.411407 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:19:49.457233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.456220 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:49.457233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.456702 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:19:49.517095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.514471 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 12:19:49.535293 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.534586 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 12:19:49.535489 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.535378 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:19:49.535489 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.535442 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:19:49.607101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.588180 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:49.621323 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.608269 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:19:49.738999 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.730732 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:49.738999 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.731448 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:19:49.746111 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.739389 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr"] Oct 30 12:19:49.802184 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.798899 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:49.898542 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.898507 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:19:49.898863 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.898845 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:19:49.913341 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.913286 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:49.944127 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.934853 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:19:49.945012 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.944986 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr" Oct 30 12:19:49.962856 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.952933 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 12:19:49.962856 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.952995 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" podUID="4ad41b97-a567-4fc6-8d54-842e1547bfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 12:19:49.973421 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.963486 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5649cd\"" Oct 30 12:19:49.973421 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:49.973360 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5649cd-dockercfg-t7xct\"" Oct 30 12:19:50.085847 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.085259 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr"] Oct 30 12:19:50.378910 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.378573 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt"] Oct 30 12:19:50.421491 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.421422 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfwss\" (UniqueName: \"kubernetes.io/projected/e6ae19ef-b176-41c3-b364-09041b61bd79-kube-api-access-pfwss\") pod \"payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr\" (UID: \"e6ae19ef-b176-41c3-b364-09041b61bd79\") " pod="openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr" Oct 30 12:19:50.422054 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.422029 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ae19ef-b176-41c3-b364-09041b61bd79-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr\" (UID: \"e6ae19ef-b176-41c3-b364-09041b61bd79\") " pod="openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr" Oct 30 12:19:50.489097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.484041 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:50.557101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.556948 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 12:19:50.594688 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.594639 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ae19ef-b176-41c3-b364-09041b61bd79-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr\" (UID: \"e6ae19ef-b176-41c3-b364-09041b61bd79\") " pod="openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr" Oct 30 12:19:50.595171 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.595143 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfwss\" (UniqueName: \"kubernetes.io/projected/e6ae19ef-b176-41c3-b364-09041b61bd79-kube-api-access-pfwss\") pod \"payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr\" (UID: \"e6ae19ef-b176-41c3-b364-09041b61bd79\") " pod="openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr" Oct 30 12:19:50.628539 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.628500 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ae19ef-b176-41c3-b364-09041b61bd79-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr\" (UID: \"e6ae19ef-b176-41c3-b364-09041b61bd79\") " pod="openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr" Oct 30 12:19:50.648452 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.648347 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:50.713198 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.712718 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:19:50.713511 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.713466 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" podUID="9d42ea5c-ec5b-433f-887a-76d2b8a68f46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:50.761094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.747698 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:50.761094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.748088 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:50.853157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.853121 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt"] Oct 30 12:19:50.853607 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.853585 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt" Oct 30 12:19:50.856178 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.856145 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:50.953397 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.953331 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:50.999239 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.999202 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 12:19:50.999501 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:50.999471 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" podUID="4ad41b97-a567-4fc6-8d54-842e1547bfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 12:19:51.071579 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.071532 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:51.131098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.075418 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 30 12:19:51.131098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.078509 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a43dcb6-b650-498f-9346-829864c2737a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt\" (UID: \"0a43dcb6-b650-498f-9346-829864c2737a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt" Oct 30 12:19:51.131098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.112766 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b22f\" (UniqueName: \"kubernetes.io/projected/0a43dcb6-b650-498f-9346-829864c2737a-kube-api-access-6b22f\") pod \"payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt\" (UID: \"0a43dcb6-b650-498f-9346-829864c2737a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt" Oct 30 12:19:51.131098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.098609 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:51.131098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.084341 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfwss\" (UniqueName: \"kubernetes.io/projected/e6ae19ef-b176-41c3-b364-09041b61bd79-kube-api-access-pfwss\") pod \"payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr\" (UID: \"e6ae19ef-b176-41c3-b364-09041b61bd79\") " pod="openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr" Oct 30 12:19:51.131098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.113299 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr" Oct 30 12:19:51.221191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.215958 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a43dcb6-b650-498f-9346-829864c2737a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt\" (UID: \"0a43dcb6-b650-498f-9346-829864c2737a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt" Oct 30 12:19:51.221191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.216169 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b22f\" (UniqueName: \"kubernetes.io/projected/0a43dcb6-b650-498f-9346-829864c2737a-kube-api-access-6b22f\") pod \"payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt\" (UID: \"0a43dcb6-b650-498f-9346-829864c2737a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt" Oct 30 12:19:51.221191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.217714 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a43dcb6-b650-498f-9346-829864c2737a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt\" (UID: \"0a43dcb6-b650-498f-9346-829864c2737a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt" Oct 30 12:19:51.386600 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.386558 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b22f\" (UniqueName: \"kubernetes.io/projected/0a43dcb6-b650-498f-9346-829864c2737a-kube-api-access-6b22f\") pod \"payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt\" (UID: \"0a43dcb6-b650-498f-9346-829864c2737a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt" Oct 30 12:19:51.483320 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.483200 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": context deadline exceeded" Oct 30 12:19:51.496586 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.496517 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:51.602428 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.602380 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt" Oct 30 12:19:51.886343 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.762781 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 12:19:51.886343 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.791531 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:19:51.886343 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.827416 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:19:51.886343 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.828298 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:51.956140 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.935671 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 12:19:51.967739 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.967690 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:51.976870 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:51.976836 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:19:52.001534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:52.001115 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:19:52.002361 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:52.002320 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 12:19:52.002477 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:52.002390 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" podUID="4ad41b97-a567-4fc6-8d54-842e1547bfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 12:19:52.021794 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:52.005815 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Oct 30 12:19:52.333219 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:52.020225 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:19:52.333790 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:52.204999 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:52.408626 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:52.408595 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:19:52.544467 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:52.543956 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 12:19:52.670701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:52.664836 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:52.695957 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:52.695925 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:19:52.796092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:52.792835 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:52.797953 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:52.797917 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:19:52.857365 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:52.857301 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt"] Oct 30 12:19:52.911946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:52.910394 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:19:52.916985 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:52.916956 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:19:52.948719 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:52.948684 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 12:19:52.948896 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:52.948749 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" podUID="4ad41b97-a567-4fc6-8d54-842e1547bfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 12:19:53.088582 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:19:53.088553 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ae19ef_b176_41c3_b364_09041b61bd79.slice/crio-996e8caf19f42cfe5ba52bf5aa254c8d525e921d33b88fa465e1cbe030cb4fe3 WatchSource:0}: Error finding container 996e8caf19f42cfe5ba52bf5aa254c8d525e921d33b88fa465e1cbe030cb4fe3: Status 404 returned error can't find the container with id 996e8caf19f42cfe5ba52bf5aa254c8d525e921d33b88fa465e1cbe030cb4fe3 Oct 30 12:19:53.094267 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:53.094237 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr"] Oct 30 12:19:53.308870 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:53.308830 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr" event={"ID":"e6ae19ef-b176-41c3-b364-09041b61bd79","Type":"ContainerStarted","Data":"e5985f1c72124d1b5dfd6c2d22ac2461c7f7b13cfa91ab9802122037d34e29de"} Oct 30 12:19:53.309100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:53.308905 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr" event={"ID":"e6ae19ef-b176-41c3-b364-09041b61bd79","Type":"ContainerStarted","Data":"996e8caf19f42cfe5ba52bf5aa254c8d525e921d33b88fa465e1cbe030cb4fe3"} Oct 30 12:19:53.309100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:53.308955 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr" Oct 30 12:19:53.310906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:53.310875 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt" event={"ID":"0a43dcb6-b650-498f-9346-829864c2737a","Type":"ContainerStarted","Data":"19b51fc1e57bef525d1fe02ec851a627fff54e3f40fe7a0bdcedb13266313e59"} Oct 30 12:19:53.311031 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:53.310928 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt" event={"ID":"0a43dcb6-b650-498f-9346-829864c2737a","Type":"ContainerStarted","Data":"4b306710601bcc78e9b8b9a73ded1a29e46ebd00921068b8f4cb964071a72cec"} Oct 30 12:19:53.349511 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:53.349457 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr" podStartSLOduration=4.349433962 podStartE2EDuration="4.349433962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:53.348177099 +0000 UTC m=+1544.382315497" watchObservedRunningTime="2026-10-30 12:19:53.349433962 +0000 UTC m=+1544.383572425" Oct 30 12:19:53.389058 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:53.389006 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt" podStartSLOduration=4.388990287 podStartE2EDuration="4.388990287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:53.38640702 +0000 UTC m=+1544.420545417" watchObservedRunningTime="2026-10-30 12:19:53.388990287 +0000 UTC m=+1544.423129026" Oct 30 12:19:53.605734 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:19:53.605640 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:19:53.952453 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:53.952412 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 12:19:53.952796 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:53.952756 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" podUID="4ad41b97-a567-4fc6-8d54-842e1547bfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 12:19:54.343650 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:54.343550 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt" Oct 30 12:19:55.111745 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:55.111707 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:55.112219 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:55.112195 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:19:55.113204 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:19:55.113182 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:20:01.017836 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:01.017788 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv"] Oct 30 12:20:01.022586 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:01.022559 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" Oct 30 12:20:01.049216 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:01.049182 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5649cd-serving-cert\"" Oct 30 12:20:01.054916 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:01.054876 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv"] Oct 30 12:20:01.171593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:01.171558 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2nht\" (UniqueName: \"kubernetes.io/projected/8a8befff-f988-4749-948f-0f4e0ca2ff1e-kube-api-access-b2nht\") pod \"maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv\" (UID: \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" Oct 30 12:20:01.171805 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:01.171638 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8befff-f988-4749-948f-0f4e0ca2ff1e-metrics-certs\") pod \"maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv\" (UID: \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" Oct 30 12:20:01.171805 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:01.171731 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8befff-f988-4749-948f-0f4e0ca2ff1e-openshift-service-ca\") pod \"maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv\" (UID: \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" Oct 30 12:20:01.171931 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:01.171912 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8befff-f988-4749-948f-0f4e0ca2ff1e-maas-api-tls\") pod \"maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv\" (UID: \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" Oct 30 12:20:01.286147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:01.278156 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2nht\" (UniqueName: \"kubernetes.io/projected/8a8befff-f988-4749-948f-0f4e0ca2ff1e-kube-api-access-b2nht\") pod \"maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv\" (UID: \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" Oct 30 12:20:01.286147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:01.278254 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8befff-f988-4749-948f-0f4e0ca2ff1e-metrics-certs\") pod \"maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv\" (UID: \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" Oct 30 12:20:01.286147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:01.278412 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8befff-f988-4749-948f-0f4e0ca2ff1e-openshift-service-ca\") pod \"maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv\" (UID: \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" Oct 30 12:20:01.286147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:01.278595 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8befff-f988-4749-948f-0f4e0ca2ff1e-maas-api-tls\") pod \"maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv\" (UID: \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" Oct 30 12:20:01.298092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:01.293977 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8befff-f988-4749-948f-0f4e0ca2ff1e-openshift-service-ca\") pod \"maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv\" (UID: \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" Oct 30 12:20:01.298092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:01.294721 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8befff-f988-4749-948f-0f4e0ca2ff1e-metrics-certs\") pod \"maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv\" (UID: \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" Oct 30 12:20:01.311096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:01.310738 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8befff-f988-4749-948f-0f4e0ca2ff1e-maas-api-tls\") pod \"maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv\" (UID: \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" Oct 30 12:20:01.334954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:01.334890 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2nht\" (UniqueName: \"kubernetes.io/projected/8a8befff-f988-4749-948f-0f4e0ca2ff1e-kube-api-access-b2nht\") pod \"maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv\" (UID: \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" Oct 30 12:20:01.344107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:01.339366 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" Oct 30 12:20:01.655405 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:01.655376 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv"] Oct 30 12:20:01.656441 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:20:01.656413 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8befff_f988_4749_948f_0f4e0ca2ff1e.slice/crio-082ef7f678b36b29be552dd893590085874a7df62bdbcd59f779e6e04e4a5644 WatchSource:0}: Error finding container 082ef7f678b36b29be552dd893590085874a7df62bdbcd59f779e6e04e4a5644: Status 404 returned error can't find the container with id 082ef7f678b36b29be552dd893590085874a7df62bdbcd59f779e6e04e4a5644 Oct 30 12:20:02.458483 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.458396 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" event={"ID":"8a8befff-f988-4749-948f-0f4e0ca2ff1e","Type":"ContainerStarted","Data":"24852d19c7e0e03095e5a766f51b059c389a4b8156d91b62f3e36f124d13a78b"} Oct 30 12:20:02.458483 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.458482 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" event={"ID":"8a8befff-f988-4749-948f-0f4e0ca2ff1e","Type":"ContainerStarted","Data":"082ef7f678b36b29be552dd893590085874a7df62bdbcd59f779e6e04e4a5644"} Oct 30 12:20:02.458894 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.458773 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" Oct 30 12:20:02.503313 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.503254 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" podStartSLOduration=1.9975084509999999 podStartE2EDuration="2.503239009s" totalImagesPullingTime="505.730558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:02.502389298 +0000 UTC m=+1553.536527696" watchObservedRunningTime="2026-10-30 12:20:02.503239009 +0000 UTC m=+1553.537377405" Oct 30 12:20:02.763144 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.761446 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml"] Oct 30 12:20:02.803768 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.803716 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.806603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.806562 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml"] Oct 30 12:20:02.860374 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.860327 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.860594 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.860446 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwk7n\" (UniqueName: \"kubernetes.io/projected/1d5f9e36-e943-405f-9f53-6ffb08d31c67-kube-api-access-pwk7n\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.860594 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.860559 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.860713 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.860600 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.860772 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.860718 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.860821 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.860780 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.860991 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.860969 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.861036 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.861026 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.861204 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.861184 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.961955 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.961916 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.962209 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.962014 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.962209 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.962042 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.962209 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.962117 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.962209 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.962158 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.962209 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.962184 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwk7n\" (UniqueName: \"kubernetes.io/projected/1d5f9e36-e943-405f-9f53-6ffb08d31c67-kube-api-access-pwk7n\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.962505 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.962392 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.962505 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.962429 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.962505 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.962492 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.962676 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.962496 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.962676 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.962618 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.962785 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.962697 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.962845 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.962828 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.962898 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.962832 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.964519 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.964494 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.964960 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.964938 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.985117 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.985063 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:02.985394 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:02.985367 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwk7n\" (UniqueName: \"kubernetes.io/projected/1d5f9e36-e943-405f-9f53-6ffb08d31c67-kube-api-access-pwk7n\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:03.132971 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:03.132882 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:03.460225 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:20:03.460186 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5f9e36_e943_405f_9f53_6ffb08d31c67.slice/crio-94e3b3fcd127465e6beb925a006f71826a9e12f51845fdf98d47336fce1bf50b WatchSource:0}: Error finding container 94e3b3fcd127465e6beb925a006f71826a9e12f51845fdf98d47336fce1bf50b: Status 404 returned error can't find the container with id 94e3b3fcd127465e6beb925a006f71826a9e12f51845fdf98d47336fce1bf50b Oct 30 12:20:03.460730 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:03.460704 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml"] Oct 30 12:20:03.463356 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:03.463314 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:03.463477 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:03.463391 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:03.463477 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:03.463422 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:04.351676 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:04.351642 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr" Oct 30 12:20:04.486929 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:04.486892 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" event={"ID":"1d5f9e36-e943-405f-9f53-6ffb08d31c67","Type":"ContainerStarted","Data":"5bd0df2ad7d91628adc626180a351c50c0105e65f55437af0e794590526f2c20"} Oct 30 12:20:04.487370 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:04.486976 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" event={"ID":"1d5f9e36-e943-405f-9f53-6ffb08d31c67","Type":"ContainerStarted","Data":"94e3b3fcd127465e6beb925a006f71826a9e12f51845fdf98d47336fce1bf50b"} Oct 30 12:20:04.545749 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:04.545683 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" podStartSLOduration=2.5456672830000002 podStartE2EDuration="2.545667283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:04.543312687 +0000 UTC m=+1555.577451084" watchObservedRunningTime="2026-10-30 12:20:04.545667283 +0000 UTC m=+1555.579805748" Oct 30 12:20:05.144095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:05.143517 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 12:20:05.144095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:05.143590 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" podUID="1d5f9e36-e943-405f-9f53-6ffb08d31c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 12:20:05.158106 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:05.150004 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:05.353655 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:05.353622 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt" Oct 30 12:20:06.133550 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:06.133511 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 12:20:06.134000 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:06.133566 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" podUID="1d5f9e36-e943-405f-9f53-6ffb08d31c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 12:20:07.133377 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:07.133335 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 12:20:07.133568 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:07.133417 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" podUID="1d5f9e36-e943-405f-9f53-6ffb08d31c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 12:20:08.138098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:08.136554 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 12:20:08.138098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:08.136623 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" podUID="1d5f9e36-e943-405f-9f53-6ffb08d31c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 12:20:08.474245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:08.474211 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" Oct 30 12:20:09.140347 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:09.140284 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 12:20:09.140842 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:09.140373 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" podUID="1d5f9e36-e943-405f-9f53-6ffb08d31c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 12:20:09.389683 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:09.389372 4468 scope.go:122] "RemoveContainer" containerID="e6534cde6cddf7149fca7024cb021cf9bd9e77567ec13507ee9811583ba18bac" Oct 30 12:20:10.134062 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:10.134011 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 12:20:10.134275 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:10.134115 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" podUID="1d5f9e36-e943-405f-9f53-6ffb08d31c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 12:20:11.140147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:11.140108 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 12:20:11.140898 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:11.140724 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" podUID="1d5f9e36-e943-405f-9f53-6ffb08d31c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 12:20:12.139040 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:12.139003 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:12.139345 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:12.139319 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:12.140242 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:12.140223 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:20:22.798856 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:22.798593 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9"] Oct 30 12:20:22.832840 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:22.819003 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:22.832840 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:22.821730 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9"] Oct 30 12:20:22.839627 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:22.839595 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7566f5-openshift-default-dockercfg-sxs8t\"" Oct 30 12:20:22.952090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:22.949960 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istiod-ca-cert\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:22.952090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:22.950039 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjwqw\" (UniqueName: \"kubernetes.io/projected/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-kube-api-access-qjwqw\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:22.952090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:22.952003 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-workload-certs\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:22.952090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:22.952057 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-workload-socket\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:22.952090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:22.952103 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-data\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:22.952520 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:22.952204 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-envoy\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:22.952520 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:22.952268 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-credential-socket\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:22.952520 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:22.952316 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-podinfo\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:22.952520 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:22.952346 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-token\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.068141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:23.067330 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-podinfo\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.068141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:23.067388 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-token\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.068141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:23.067520 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istiod-ca-cert\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.068141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:23.067557 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjwqw\" (UniqueName: \"kubernetes.io/projected/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-kube-api-access-qjwqw\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.068141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:23.067700 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-workload-certs\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.068141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:23.067746 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-workload-socket\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.068141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:23.067779 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-data\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.068141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:23.067914 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-envoy\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.068141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:23.067981 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-credential-socket\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.068759 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:23.068693 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-credential-socket\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.093536 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:23.093488 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-workload-certs\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.147165 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:23.094151 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-data\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.147165 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:23.094445 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-workload-socket\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.147165 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:23.095230 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istiod-ca-cert\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.186496 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:23.186457 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjwqw\" (UniqueName: \"kubernetes.io/projected/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-kube-api-access-qjwqw\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.187277 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:23.187249 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-envoy\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.188105 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:23.188047 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-podinfo\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.201850 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:23.201169 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-token\") pod \"e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.483093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:23.482615 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:23.676106 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:20:23.675362 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:20:24.024492 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:20:24.024447 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8278ddb9_839e_4d62_b0b0_ba9cf036ae10.slice/crio-253f841502d77edff6ca87159c51682dd0a8d21d17772c26dfbda91290e293f6 WatchSource:0}: Error finding container 253f841502d77edff6ca87159c51682dd0a8d21d17772c26dfbda91290e293f6: Status 404 returned error can't find the container with id 253f841502d77edff6ca87159c51682dd0a8d21d17772c26dfbda91290e293f6 Oct 30 12:20:24.032736 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:24.032684 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:24.032996 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:24.032969 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:24.033215 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:24.033193 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:24.076151 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:24.076123 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9"] Oct 30 12:20:24.767248 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:24.767199 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" event={"ID":"8278ddb9-839e-4d62-b0b0-ba9cf036ae10","Type":"ContainerStarted","Data":"eb7fbf606df0f6a6e7721840290827991f20577d169918cbba8279670034acd7"} Oct 30 12:20:24.767543 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:24.767518 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" event={"ID":"8278ddb9-839e-4d62-b0b0-ba9cf036ae10","Type":"ContainerStarted","Data":"253f841502d77edff6ca87159c51682dd0a8d21d17772c26dfbda91290e293f6"} Oct 30 12:20:24.827574 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:24.827494 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" podStartSLOduration=2.827470061 podStartE2EDuration="2.827470061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:24.821624412 +0000 UTC m=+1575.855762809" watchObservedRunningTime="2026-10-30 12:20:24.827470061 +0000 UTC m=+1575.861608457" Oct 30 12:20:25.484045 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:25.484001 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:25.491256 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:25.491215 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:25.781630 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:25.781543 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:25.782770 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:25.782744 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:20:29.417699 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:29.417656 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8"] Oct 30 12:20:29.454596 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:29.454497 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8"] Oct 30 12:20:29.455303 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:29.455219 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8" Oct 30 12:20:29.500291 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:29.478477 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7566f5\"" Oct 30 12:20:29.574092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:29.567002 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7566f5-dockercfg-nsvjc\"" Oct 30 12:20:29.575596 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:29.575545 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61651192-0a38-49aa-8b90-20a364a9761b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8\" (UID: \"61651192-0a38-49aa-8b90-20a364a9761b\") " pod="openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8" Oct 30 12:20:29.577578 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:29.577533 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6pxs\" (UniqueName: \"kubernetes.io/projected/61651192-0a38-49aa-8b90-20a364a9761b-kube-api-access-b6pxs\") pod \"payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8\" (UID: \"61651192-0a38-49aa-8b90-20a364a9761b\") " pod="openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8" Oct 30 12:20:29.679624 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:29.679523 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6pxs\" (UniqueName: \"kubernetes.io/projected/61651192-0a38-49aa-8b90-20a364a9761b-kube-api-access-b6pxs\") pod \"payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8\" (UID: \"61651192-0a38-49aa-8b90-20a364a9761b\") " pod="openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8" Oct 30 12:20:29.679804 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:29.679721 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61651192-0a38-49aa-8b90-20a364a9761b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8\" (UID: \"61651192-0a38-49aa-8b90-20a364a9761b\") " pod="openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8" Oct 30 12:20:29.680481 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:29.680459 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61651192-0a38-49aa-8b90-20a364a9761b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8\" (UID: \"61651192-0a38-49aa-8b90-20a364a9761b\") " pod="openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8" Oct 30 12:20:29.702294 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:29.702258 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6pxs\" (UniqueName: \"kubernetes.io/projected/61651192-0a38-49aa-8b90-20a364a9761b-kube-api-access-b6pxs\") pod \"payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8\" (UID: \"61651192-0a38-49aa-8b90-20a364a9761b\") " pod="openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8" Oct 30 12:20:29.918056 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:29.918021 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8" Oct 30 12:20:30.227519 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:30.227486 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8"] Oct 30 12:20:30.227660 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:20:30.227551 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61651192_0a38_49aa_8b90_20a364a9761b.slice/crio-e673b74a24a09617604e272321bf463ff197ca37fbf294b6b097c9694a16bded WatchSource:0}: Error finding container e673b74a24a09617604e272321bf463ff197ca37fbf294b6b097c9694a16bded: Status 404 returned error can't find the container with id e673b74a24a09617604e272321bf463ff197ca37fbf294b6b097c9694a16bded Oct 30 12:20:30.820343 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:30.820308 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8" event={"ID":"61651192-0a38-49aa-8b90-20a364a9761b","Type":"ContainerStarted","Data":"2126ec66266f0f11373a60d504bb51107ff580f41fb0dbae22885ab3883db769"} Oct 30 12:20:30.820779 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:30.820361 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8" event={"ID":"61651192-0a38-49aa-8b90-20a364a9761b","Type":"ContainerStarted","Data":"e673b74a24a09617604e272321bf463ff197ca37fbf294b6b097c9694a16bded"} Oct 30 12:20:30.820779 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:30.820395 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8" Oct 30 12:20:30.859849 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:30.859785 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8" podStartSLOduration=1.859771238 podStartE2EDuration="1.859771238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:30.858399405 +0000 UTC m=+1581.892537800" watchObservedRunningTime="2026-10-30 12:20:30.859771238 +0000 UTC m=+1581.893909634" Oct 30 12:20:38.175135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.175098 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb"] Oct 30 12:20:38.180545 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.180523 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb" Oct 30 12:20:38.201114 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.201080 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb"] Oct 30 12:20:38.207007 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.206970 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-82299a\"" Oct 30 12:20:38.207187 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.207047 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-82299a-dockercfg-cgxhd\"" Oct 30 12:20:38.211015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.210956 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e384fc75-3e1d-466d-8794-6e43f37e6e22-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb\" (UID: \"e384fc75-3e1d-466d-8794-6e43f37e6e22\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb" Oct 30 12:20:38.211312 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.211288 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxd6j\" (UniqueName: \"kubernetes.io/projected/e384fc75-3e1d-466d-8794-6e43f37e6e22-kube-api-access-hxd6j\") pod \"payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb\" (UID: \"e384fc75-3e1d-466d-8794-6e43f37e6e22\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb" Oct 30 12:20:38.320340 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.320296 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxd6j\" (UniqueName: \"kubernetes.io/projected/e384fc75-3e1d-466d-8794-6e43f37e6e22-kube-api-access-hxd6j\") pod \"payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb\" (UID: \"e384fc75-3e1d-466d-8794-6e43f37e6e22\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb" Oct 30 12:20:38.320958 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.320922 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e384fc75-3e1d-466d-8794-6e43f37e6e22-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb\" (UID: \"e384fc75-3e1d-466d-8794-6e43f37e6e22\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb" Oct 30 12:20:38.322955 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.322927 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e384fc75-3e1d-466d-8794-6e43f37e6e22-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb\" (UID: \"e384fc75-3e1d-466d-8794-6e43f37e6e22\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb" Oct 30 12:20:38.364090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.359532 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxd6j\" (UniqueName: \"kubernetes.io/projected/e384fc75-3e1d-466d-8794-6e43f37e6e22-kube-api-access-hxd6j\") pod \"payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb\" (UID: \"e384fc75-3e1d-466d-8794-6e43f37e6e22\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb" Oct 30 12:20:38.525021 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.524846 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb" Oct 30 12:20:38.590836 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.586308 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8"] Oct 30 12:20:38.599850 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.599814 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8"] Oct 30 12:20:38.600058 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.599957 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8" Oct 30 12:20:38.654354 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.654311 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d42735-89ea-4146-853e-2e3382b4547a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8\" (UID: \"82d42735-89ea-4146-853e-2e3382b4547a\") " pod="openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8" Oct 30 12:20:38.654525 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.654411 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s874z\" (UniqueName: \"kubernetes.io/projected/82d42735-89ea-4146-853e-2e3382b4547a-kube-api-access-s874z\") pod \"payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8\" (UID: \"82d42735-89ea-4146-853e-2e3382b4547a\") " pod="openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8" Oct 30 12:20:38.756313 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.756272 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d42735-89ea-4146-853e-2e3382b4547a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8\" (UID: \"82d42735-89ea-4146-853e-2e3382b4547a\") " pod="openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8" Oct 30 12:20:38.756523 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.756370 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s874z\" (UniqueName: \"kubernetes.io/projected/82d42735-89ea-4146-853e-2e3382b4547a-kube-api-access-s874z\") pod \"payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8\" (UID: \"82d42735-89ea-4146-853e-2e3382b4547a\") " pod="openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8" Oct 30 12:20:38.757116 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.757094 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d42735-89ea-4146-853e-2e3382b4547a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8\" (UID: \"82d42735-89ea-4146-853e-2e3382b4547a\") " pod="openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8" Oct 30 12:20:38.779436 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.779351 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s874z\" (UniqueName: \"kubernetes.io/projected/82d42735-89ea-4146-853e-2e3382b4547a-kube-api-access-s874z\") pod \"payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8\" (UID: \"82d42735-89ea-4146-853e-2e3382b4547a\") " pod="openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8" Oct 30 12:20:38.897489 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:20:38.897455 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode384fc75_3e1d_466d_8794_6e43f37e6e22.slice/crio-8dd61f9e385b20daea45020c77d0b95533358457e47bf64f166d62ada7530152 WatchSource:0}: Error finding container 8dd61f9e385b20daea45020c77d0b95533358457e47bf64f166d62ada7530152: Status 404 returned error can't find the container with id 8dd61f9e385b20daea45020c77d0b95533358457e47bf64f166d62ada7530152 Oct 30 12:20:38.898245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.898218 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb"] Oct 30 12:20:38.933162 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.933124 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8" Oct 30 12:20:38.943361 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:38.943309 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb" event={"ID":"e384fc75-3e1d-466d-8794-6e43f37e6e22","Type":"ContainerStarted","Data":"8dd61f9e385b20daea45020c77d0b95533358457e47bf64f166d62ada7530152"} Oct 30 12:20:39.256604 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:20:39.256571 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d42735_89ea_4146_853e_2e3382b4547a.slice/crio-71f37c71c0d290d45b248977e586ca5b0d06ced6f4f14a428bceb562b2312728 WatchSource:0}: Error finding container 71f37c71c0d290d45b248977e586ca5b0d06ced6f4f14a428bceb562b2312728: Status 404 returned error can't find the container with id 71f37c71c0d290d45b248977e586ca5b0d06ced6f4f14a428bceb562b2312728 Oct 30 12:20:39.256604 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:39.256593 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8"] Oct 30 12:20:39.952483 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:39.952444 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb" event={"ID":"e384fc75-3e1d-466d-8794-6e43f37e6e22","Type":"ContainerStarted","Data":"040f3c54dec2d4aa3dc1e135074a03ff66b4febe774c7b5572ce18c700cb3253"} Oct 30 12:20:39.952688 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:39.952513 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb" Oct 30 12:20:39.954124 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:39.954093 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8" event={"ID":"82d42735-89ea-4146-853e-2e3382b4547a","Type":"ContainerStarted","Data":"9f32ed83bb00095fdff6ed4bfe82e85eee31aaf5fb7804e5d3e1a680a169b147"} Oct 30 12:20:39.954242 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:39.954145 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8" event={"ID":"82d42735-89ea-4146-853e-2e3382b4547a","Type":"ContainerStarted","Data":"71f37c71c0d290d45b248977e586ca5b0d06ced6f4f14a428bceb562b2312728"} Oct 30 12:20:39.954242 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:39.954167 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8" Oct 30 12:20:40.004955 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:40.004890 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb" podStartSLOduration=2.004874945 podStartE2EDuration="2.004874945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:40.003063129 +0000 UTC m=+1591.037201506" watchObservedRunningTime="2026-10-30 12:20:40.004874945 +0000 UTC m=+1591.039013341" Oct 30 12:20:40.045625 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:40.045563 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8" podStartSLOduration=2.045545527 podStartE2EDuration="2.045545527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:40.042724755 +0000 UTC m=+1591.076863150" watchObservedRunningTime="2026-10-30 12:20:40.045545527 +0000 UTC m=+1591.079683922" Oct 30 12:20:41.831244 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:41.831213 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8" Oct 30 12:20:47.820102 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:47.820027 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5"] Oct 30 12:20:47.826181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:47.826145 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:47.842307 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:47.842273 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-060f41f2-openshift-default-dockercfg-n2q4d\"" Oct 30 12:20:47.864816 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:47.864777 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5"] Oct 30 12:20:47.919814 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:47.919770 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs"] Oct 30 12:20:47.926683 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:47.926645 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:47.967266 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:47.967205 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs"] Oct 30 12:20:47.975044 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:47.975005 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tssj7\" (UniqueName: \"kubernetes.io/projected/2d509e5c-3337-4787-9734-3749069257ec-kube-api-access-tssj7\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:47.975210 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:47.975082 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-workload-socket\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:47.975210 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:47.975169 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d509e5c-3337-4787-9734-3749069257ec-istiod-ca-cert\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:47.975351 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:47.975276 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-istio-envoy\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:47.975351 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:47.975322 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-istio-data\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:47.975468 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:47.975378 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-workload-certs\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:47.975468 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:47.975412 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d509e5c-3337-4787-9734-3749069257ec-istio-token\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:47.975586 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:47.975515 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-credential-socket\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:47.975586 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:47.975566 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d509e5c-3337-4787-9734-3749069257ec-istio-podinfo\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.077245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.077123 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.077245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.077186 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-credential-socket\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.077536 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.077302 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d509e5c-3337-4787-9734-3749069257ec-istio-podinfo\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.077536 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.077364 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.077536 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.077429 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.077668 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.077595 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tssj7\" (UniqueName: \"kubernetes.io/projected/2d509e5c-3337-4787-9734-3749069257ec-kube-api-access-tssj7\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.077668 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.077626 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-credential-socket\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.077668 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.077641 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rthkl\" (UniqueName: \"kubernetes.io/projected/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-kube-api-access-rthkl\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.077826 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.077751 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-workload-socket\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.077826 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.077815 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.077910 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.077896 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d509e5c-3337-4787-9734-3749069257ec-istiod-ca-cert\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.078024 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.078002 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.078086 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.078056 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.078151 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.078135 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-workload-socket\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.078189 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.078171 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-istio-envoy\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.078254 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.078227 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.078313 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.078286 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-istio-data\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.078450 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.078427 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-workload-certs\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.078519 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.078465 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.078519 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.078507 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d509e5c-3337-4787-9734-3749069257ec-istio-token\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.078630 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.078557 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-istio-data\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.078684 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.078660 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d509e5c-3337-4787-9734-3749069257ec-istiod-ca-cert\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.078793 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.078768 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-workload-certs\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.080413 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.080386 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d509e5c-3337-4787-9734-3749069257ec-istio-podinfo\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.080735 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.080713 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-istio-envoy\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.101389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.101353 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tssj7\" (UniqueName: \"kubernetes.io/projected/2d509e5c-3337-4787-9734-3749069257ec-kube-api-access-tssj7\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.115747 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.108401 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d509e5c-3337-4787-9734-3749069257ec-istio-token\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.153386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.152326 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:48.200221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.180045 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.200221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.180202 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rthkl\" (UniqueName: \"kubernetes.io/projected/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-kube-api-access-rthkl\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.200221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.180263 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.200221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.180358 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.200221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.180396 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.200221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.180439 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.200221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.180495 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.200221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.180564 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.200221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.180684 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.200221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.181504 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.200221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.181785 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.200221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.182386 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.201033 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.200512 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.207625 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.201245 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.217092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.209393 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.225234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.217814 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rthkl\" (UniqueName: \"kubernetes.io/projected/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-kube-api-access-rthkl\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.232996 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.225697 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.248665 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.248580 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.299121 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.285957 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:48.865050 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.865006 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5"] Oct 30 12:20:48.880169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.880131 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:48.880310 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.880228 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:48.880310 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:48.880296 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:49.046091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:49.040952 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:49.046091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:49.041049 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:49.046091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:49.042913 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:49.046091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:49.045185 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs"] Oct 30 12:20:49.082368 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:49.081427 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" event={"ID":"53efa6ea-7b24-4df7-b9b3-46ab86493fc1","Type":"ContainerStarted","Data":"a55b939c7a0180bde9fb2b7519c6d077c94332e5167835c39e710b8f95ce30f0"} Oct 30 12:20:49.083724 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:49.083688 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" event={"ID":"2d509e5c-3337-4787-9734-3749069257ec","Type":"ContainerStarted","Data":"8684e3275ec7fa3d337330b8aaa1eae89915aed3e6a6525d32b29ddeb80b9971"} Oct 30 12:20:49.083859 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:49.083754 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" event={"ID":"2d509e5c-3337-4787-9734-3749069257ec","Type":"ContainerStarted","Data":"5848fe1518f4714d7d48d0cf4431f79c6dd876bb04401d881e566c32774586a1"} Oct 30 12:20:49.134228 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:49.134182 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" podStartSLOduration=2.134166858 podStartE2EDuration="2.134166858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:49.128788184 +0000 UTC m=+1600.162926579" watchObservedRunningTime="2026-10-30 12:20:49.134166858 +0000 UTC m=+1600.168305255" Oct 30 12:20:49.155089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:49.155038 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:49.156393 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:49.156357 4468 patch_prober.go:29] interesting pod/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 12:20:49.156597 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:49.156568 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" podUID="2d509e5c-3337-4787-9734-3749069257ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 12:20:50.094675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:50.094634 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" event={"ID":"53efa6ea-7b24-4df7-b9b3-46ab86493fc1","Type":"ContainerStarted","Data":"675403ae1d4426df2a949bddfa0468e39be553afe17710d97c22057e5ba0b1f8"} Oct 30 12:20:50.158499 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:50.158452 4468 patch_prober.go:29] interesting pod/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 12:20:50.158680 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:50.158516 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" podUID="2d509e5c-3337-4787-9734-3749069257ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 12:20:50.178884 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:50.178813 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" podStartSLOduration=3.178798264 podStartE2EDuration="3.178798264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:50.177592078 +0000 UTC m=+1601.211730474" watchObservedRunningTime="2026-10-30 12:20:50.178798264 +0000 UTC m=+1601.212936660" Oct 30 12:20:50.286921 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:50.286877 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:50.289109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:50.289047 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 12:20:50.289426 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:50.289395 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" podUID="53efa6ea-7b24-4df7-b9b3-46ab86493fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 12:20:50.965424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:50.965382 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb" Oct 30 12:20:50.965930 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:50.965908 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8" Oct 30 12:20:51.158092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:51.157669 4468 patch_prober.go:29] interesting pod/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 12:20:51.158092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:51.157749 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" podUID="2d509e5c-3337-4787-9734-3749069257ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 12:20:51.325894 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:51.320177 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 12:20:51.325894 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:51.320248 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" podUID="53efa6ea-7b24-4df7-b9b3-46ab86493fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 12:20:52.156093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:52.155566 4468 patch_prober.go:29] interesting pod/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 12:20:52.156093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:52.155632 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" podUID="2d509e5c-3337-4787-9734-3749069257ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 12:20:52.287626 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:52.287569 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 12:20:52.288021 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:52.287655 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" podUID="53efa6ea-7b24-4df7-b9b3-46ab86493fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 12:20:53.154092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:53.153576 4468 patch_prober.go:29] interesting pod/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 12:20:53.154092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:53.153648 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" podUID="2d509e5c-3337-4787-9734-3749069257ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 12:20:53.287444 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:53.287402 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 12:20:53.287742 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:53.287713 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" podUID="53efa6ea-7b24-4df7-b9b3-46ab86493fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 12:20:53.782289 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:20:53.781888 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:20:54.155148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:54.153591 4468 patch_prober.go:29] interesting pod/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 12:20:54.155148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:54.153657 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" podUID="2d509e5c-3337-4787-9734-3749069257ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 12:20:54.297522 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:54.297479 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 12:20:54.298189 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:54.298150 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" podUID="53efa6ea-7b24-4df7-b9b3-46ab86493fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 12:20:55.156273 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:55.156230 4468 patch_prober.go:29] interesting pod/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 12:20:55.156582 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:55.156548 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" podUID="2d509e5c-3337-4787-9734-3749069257ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 12:20:55.287713 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:55.287666 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 12:20:55.287892 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:55.287735 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" podUID="53efa6ea-7b24-4df7-b9b3-46ab86493fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 12:20:56.214649 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:56.214610 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:56.215875 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:56.215793 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:56.217654 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:56.217629 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:20:56.287119 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:56.287016 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 12:20:56.287311 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:56.287126 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" podUID="53efa6ea-7b24-4df7-b9b3-46ab86493fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 12:20:57.292015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:57.291984 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:57.292477 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:57.292458 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:20:57.293422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:20:57.293397 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:21:17.635503 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:17.635329 4468 generic.go:356] "Generic (PLEG): container finished" podID="8278ddb9-839e-4d62-b0b0-ba9cf036ae10" containerID="eb7fbf606df0f6a6e7721840290827991f20577d169918cbba8279670034acd7" exitCode=0 Oct 30 12:21:17.635503 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:17.635404 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" event={"ID":"8278ddb9-839e-4d62-b0b0-ba9cf036ae10","Type":"ContainerDied","Data":"eb7fbf606df0f6a6e7721840290827991f20577d169918cbba8279670034acd7"} Oct 30 12:21:17.636082 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:17.635908 4468 scope.go:122] "RemoveContainer" containerID="eb7fbf606df0f6a6e7721840290827991f20577d169918cbba8279670034acd7" Oct 30 12:21:17.638260 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:17.638228 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:21:17.638364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:17.638302 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:21:17.638364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:17.638337 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:21:18.648861 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:18.648824 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" event={"ID":"8278ddb9-839e-4d62-b0b0-ba9cf036ae10","Type":"ContainerStarted","Data":"987e3f9fb2335a367e6605b2132ae3e5c58c93117883c0b6391a962ff1e5719e"} Oct 30 12:21:19.484154 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:19.484109 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:21:19.484414 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:19.484229 4468 patch_prober.go:29] interesting pod/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 12:21:19.484414 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:19.484277 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" podUID="8278ddb9-839e-4d62-b0b0-ba9cf036ae10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 12:21:20.484319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:20.484280 4468 patch_prober.go:29] interesting pod/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 12:21:20.484903 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:20.484862 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" podUID="8278ddb9-839e-4d62-b0b0-ba9cf036ae10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 12:21:21.483528 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:21.483466 4468 patch_prober.go:29] interesting pod/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 12:21:21.483746 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:21.483527 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" podUID="8278ddb9-839e-4d62-b0b0-ba9cf036ae10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 12:21:22.032933 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:22.032879 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr"] Oct 30 12:21:22.063909 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:22.063871 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt"] Oct 30 12:21:22.484328 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:22.484288 4468 patch_prober.go:29] interesting pod/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 12:21:22.484529 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:22.484360 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" podUID="8278ddb9-839e-4d62-b0b0-ba9cf036ae10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 12:21:23.483559 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:23.483514 4468 patch_prober.go:29] interesting pod/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 12:21:23.483984 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:23.483573 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" podUID="8278ddb9-839e-4d62-b0b0-ba9cf036ae10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 12:21:23.602452 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:21:23.602402 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:21:24.489540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:24.489509 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:21:24.489992 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:24.489927 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:21:24.490914 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:24.490894 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:21:24.921049 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:24.921009 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94"] Oct 30 12:21:24.924518 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:24.924492 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94" Oct 30 12:21:24.939058 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:24.939024 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ebc6bb-dockercfg-jpfmp\"" Oct 30 12:21:24.940224 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:24.940179 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ebc6bb\"" Oct 30 12:21:24.944588 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:24.944558 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94"] Oct 30 12:21:25.109093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:25.101858 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4ed41c3-aa61-4f83-b933-5f22e327f054-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94\" (UID: \"a4ed41c3-aa61-4f83-b933-5f22e327f054\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94" Oct 30 12:21:25.109093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:25.101975 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pphsq\" (UniqueName: \"kubernetes.io/projected/a4ed41c3-aa61-4f83-b933-5f22e327f054-kube-api-access-pphsq\") pod \"payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94\" (UID: \"a4ed41c3-aa61-4f83-b933-5f22e327f054\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94" Oct 30 12:21:25.224627 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:25.209562 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4ed41c3-aa61-4f83-b933-5f22e327f054-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94\" (UID: \"a4ed41c3-aa61-4f83-b933-5f22e327f054\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94" Oct 30 12:21:25.224627 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:25.209696 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pphsq\" (UniqueName: \"kubernetes.io/projected/a4ed41c3-aa61-4f83-b933-5f22e327f054-kube-api-access-pphsq\") pod \"payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94\" (UID: \"a4ed41c3-aa61-4f83-b933-5f22e327f054\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94" Oct 30 12:21:25.226174 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:25.226131 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4ed41c3-aa61-4f83-b933-5f22e327f054-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94\" (UID: \"a4ed41c3-aa61-4f83-b933-5f22e327f054\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94" Oct 30 12:21:25.295697 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:25.295329 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pphsq\" (UniqueName: \"kubernetes.io/projected/a4ed41c3-aa61-4f83-b933-5f22e327f054-kube-api-access-pphsq\") pod \"payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94\" (UID: \"a4ed41c3-aa61-4f83-b933-5f22e327f054\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94" Oct 30 12:21:25.541447 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:25.541357 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94" Oct 30 12:21:25.904134 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:25.904096 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94"] Oct 30 12:21:26.766924 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:26.766880 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94" event={"ID":"a4ed41c3-aa61-4f83-b933-5f22e327f054","Type":"ContainerStarted","Data":"b1255723de1034a617790e37d103692adbb1219e1f8d177351556be6f70c3837"} Oct 30 12:21:26.767385 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:26.766954 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94" Oct 30 12:21:26.767385 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:26.766982 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94" event={"ID":"a4ed41c3-aa61-4f83-b933-5f22e327f054","Type":"ContainerStarted","Data":"98186d7729e68e5a6e6a567c7df84bc9787c458e4ed2547738b793ae093dbf8e"} Oct 30 12:21:26.812778 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:26.812711 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94" podStartSLOduration=2.8126955860000002 podStartE2EDuration="2.812695586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:21:26.811183488 +0000 UTC m=+1637.845321887" watchObservedRunningTime="2026-10-30 12:21:26.812695586 +0000 UTC m=+1637.846833982" Oct 30 12:21:27.037319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.037177 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr" podUID="e6ae19ef-b176-41c3-b364-09041b61bd79" containerName="payload-processing" containerID="cri-o://e5985f1c72124d1b5dfd6c2d22ac2461c7f7b13cfa91ab9802122037d34e29de" gracePeriod=25 Oct 30 12:21:27.064675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.064598 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt" podUID="0a43dcb6-b650-498f-9346-829864c2737a" containerName="payload-pre-processing" containerID="cri-o://19b51fc1e57bef525d1fe02ec851a627fff54e3f40fe7a0bdcedb13266313e59" gracePeriod=25 Oct 30 12:21:27.360270 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.360242 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr" Oct 30 12:21:27.364020 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.363994 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ae19ef-b176-41c3-b364-09041b61bd79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ae19ef-b176-41c3-b364-09041b61bd79-plugins-config-volume\") pod \"e6ae19ef-b176-41c3-b364-09041b61bd79\" (UID: \"e6ae19ef-b176-41c3-b364-09041b61bd79\") " Oct 30 12:21:27.364317 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.364300 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ae19ef-b176-41c3-b364-09041b61bd79-kube-api-access-pfwss\" (UniqueName: \"kubernetes.io/projected/e6ae19ef-b176-41c3-b364-09041b61bd79-kube-api-access-pfwss\") pod \"e6ae19ef-b176-41c3-b364-09041b61bd79\" (UID: \"e6ae19ef-b176-41c3-b364-09041b61bd79\") " Oct 30 12:21:27.364479 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.364450 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ae19ef-b176-41c3-b364-09041b61bd79-plugins-config-volume" pod "e6ae19ef-b176-41c3-b364-09041b61bd79" (UID: "e6ae19ef-b176-41c3-b364-09041b61bd79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:21:27.364969 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.364944 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ae19ef-b176-41c3-b364-09041b61bd79-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:21:27.366579 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.366552 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ae19ef-b176-41c3-b364-09041b61bd79-kube-api-access-pfwss" pod "e6ae19ef-b176-41c3-b364-09041b61bd79" (UID: "e6ae19ef-b176-41c3-b364-09041b61bd79"). InnerVolumeSpecName "kube-api-access-pfwss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:21:27.389914 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.389886 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt" Oct 30 12:21:27.466530 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.466486 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a43dcb6-b650-498f-9346-829864c2737a-kube-api-access-6b22f\" (UniqueName: \"kubernetes.io/projected/0a43dcb6-b650-498f-9346-829864c2737a-kube-api-access-6b22f\") pod \"0a43dcb6-b650-498f-9346-829864c2737a\" (UID: \"0a43dcb6-b650-498f-9346-829864c2737a\") " Oct 30 12:21:27.466758 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.466683 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a43dcb6-b650-498f-9346-829864c2737a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a43dcb6-b650-498f-9346-829864c2737a-plugins-config-volume\") pod \"0a43dcb6-b650-498f-9346-829864c2737a\" (UID: \"0a43dcb6-b650-498f-9346-829864c2737a\") " Oct 30 12:21:27.467275 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.467242 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a43dcb6-b650-498f-9346-829864c2737a-plugins-config-volume" pod "0a43dcb6-b650-498f-9346-829864c2737a" (UID: "0a43dcb6-b650-498f-9346-829864c2737a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:21:27.467556 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.467533 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a43dcb6-b650-498f-9346-829864c2737a-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:21:27.467613 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.467565 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfwss\" (UniqueName: \"kubernetes.io/projected/e6ae19ef-b176-41c3-b364-09041b61bd79-kube-api-access-pfwss\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:21:27.469575 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.469553 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a43dcb6-b650-498f-9346-829864c2737a-kube-api-access-6b22f" pod "0a43dcb6-b650-498f-9346-829864c2737a" (UID: "0a43dcb6-b650-498f-9346-829864c2737a"). InnerVolumeSpecName "kube-api-access-6b22f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:21:27.570298 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.570212 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b22f\" (UniqueName: \"kubernetes.io/projected/0a43dcb6-b650-498f-9346-829864c2737a-kube-api-access-6b22f\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:21:27.776651 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.776616 4468 generic.go:356] "Generic (PLEG): container finished" podID="0a43dcb6-b650-498f-9346-829864c2737a" containerID="19b51fc1e57bef525d1fe02ec851a627fff54e3f40fe7a0bdcedb13266313e59" exitCode=0 Oct 30 12:21:27.777166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.776674 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt" Oct 30 12:21:27.777166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.776697 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt" event={"ID":"0a43dcb6-b650-498f-9346-829864c2737a","Type":"ContainerDied","Data":"19b51fc1e57bef525d1fe02ec851a627fff54e3f40fe7a0bdcedb13266313e59"} Oct 30 12:21:27.777166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.776746 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt" event={"ID":"0a43dcb6-b650-498f-9346-829864c2737a","Type":"ContainerDied","Data":"4b306710601bcc78e9b8b9a73ded1a29e46ebd00921068b8f4cb964071a72cec"} Oct 30 12:21:27.777166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.776765 4468 scope.go:122] "RemoveContainer" containerID="19b51fc1e57bef525d1fe02ec851a627fff54e3f40fe7a0bdcedb13266313e59" Oct 30 12:21:27.778467 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.778440 4468 generic.go:356] "Generic (PLEG): container finished" podID="e6ae19ef-b176-41c3-b364-09041b61bd79" containerID="e5985f1c72124d1b5dfd6c2d22ac2461c7f7b13cfa91ab9802122037d34e29de" exitCode=0 Oct 30 12:21:27.778593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.778500 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr" Oct 30 12:21:27.778593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.778582 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr" event={"ID":"e6ae19ef-b176-41c3-b364-09041b61bd79","Type":"ContainerDied","Data":"e5985f1c72124d1b5dfd6c2d22ac2461c7f7b13cfa91ab9802122037d34e29de"} Oct 30 12:21:27.778681 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.778624 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr" event={"ID":"e6ae19ef-b176-41c3-b364-09041b61bd79","Type":"ContainerDied","Data":"996e8caf19f42cfe5ba52bf5aa254c8d525e921d33b88fa465e1cbe030cb4fe3"} Oct 30 12:21:27.791318 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.791295 4468 scope.go:122] "RemoveContainer" containerID="19b51fc1e57bef525d1fe02ec851a627fff54e3f40fe7a0bdcedb13266313e59" Oct 30 12:21:27.791720 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:21:27.791695 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b51fc1e57bef525d1fe02ec851a627fff54e3f40fe7a0bdcedb13266313e59\": container with ID starting with 19b51fc1e57bef525d1fe02ec851a627fff54e3f40fe7a0bdcedb13266313e59 not found: ID does not exist" containerID="19b51fc1e57bef525d1fe02ec851a627fff54e3f40fe7a0bdcedb13266313e59" Oct 30 12:21:27.791824 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.791728 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b51fc1e57bef525d1fe02ec851a627fff54e3f40fe7a0bdcedb13266313e59"} err="failed to get container status \"19b51fc1e57bef525d1fe02ec851a627fff54e3f40fe7a0bdcedb13266313e59\": rpc error: code = NotFound desc = could not find container \"19b51fc1e57bef525d1fe02ec851a627fff54e3f40fe7a0bdcedb13266313e59\": container with ID starting with 19b51fc1e57bef525d1fe02ec851a627fff54e3f40fe7a0bdcedb13266313e59 not found: ID does not exist" Oct 30 12:21:27.791824 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.791743 4468 scope.go:122] "RemoveContainer" containerID="e5985f1c72124d1b5dfd6c2d22ac2461c7f7b13cfa91ab9802122037d34e29de" Oct 30 12:21:27.802298 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.802275 4468 scope.go:122] "RemoveContainer" containerID="e5985f1c72124d1b5dfd6c2d22ac2461c7f7b13cfa91ab9802122037d34e29de" Oct 30 12:21:27.802629 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:21:27.802602 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5985f1c72124d1b5dfd6c2d22ac2461c7f7b13cfa91ab9802122037d34e29de\": container with ID starting with e5985f1c72124d1b5dfd6c2d22ac2461c7f7b13cfa91ab9802122037d34e29de not found: ID does not exist" containerID="e5985f1c72124d1b5dfd6c2d22ac2461c7f7b13cfa91ab9802122037d34e29de" Oct 30 12:21:27.802680 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.802638 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5985f1c72124d1b5dfd6c2d22ac2461c7f7b13cfa91ab9802122037d34e29de"} err="failed to get container status \"e5985f1c72124d1b5dfd6c2d22ac2461c7f7b13cfa91ab9802122037d34e29de\": rpc error: code = NotFound desc = could not find container \"e5985f1c72124d1b5dfd6c2d22ac2461c7f7b13cfa91ab9802122037d34e29de\": container with ID starting with e5985f1c72124d1b5dfd6c2d22ac2461c7f7b13cfa91ab9802122037d34e29de not found: ID does not exist" Oct 30 12:21:27.827623 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.827502 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr"] Oct 30 12:21:27.831617 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.831587 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5649cd-55985f67d8-6kckr"] Oct 30 12:21:27.863306 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.863275 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt"] Oct 30 12:21:27.867017 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:27.866987 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5649cd-7557d8867f-r97mt"] Oct 30 12:21:29.091435 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:29.091388 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a43dcb6-b650-498f-9346-829864c2737a" path="/var/lib/kubelet/pods/0a43dcb6-b650-498f-9346-829864c2737a/volumes" Oct 30 12:21:29.092083 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:29.092049 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ae19ef-b176-41c3-b364-09041b61bd79" path="/var/lib/kubelet/pods/e6ae19ef-b176-41c3-b364-09041b61bd79/volumes" Oct 30 12:21:36.649353 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:36.649305 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g"] Oct 30 12:21:37.812319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:37.812279 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94" Oct 30 12:21:40.510549 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.510508 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt"] Oct 30 12:21:40.511399 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.511377 4468 cpu_manager.go:414] "containerMap: removing container" podUID="e6ae19ef-b176-41c3-b364-09041b61bd79" containerName="payload-processing" Oct 30 12:21:40.511469 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.511400 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ae19ef-b176-41c3-b364-09041b61bd79" containerName="payload-processing" Oct 30 12:21:40.511469 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.511430 4468 cpu_manager.go:414] "containerMap: removing container" podUID="0a43dcb6-b650-498f-9346-829864c2737a" containerName="payload-pre-processing" Oct 30 12:21:40.511469 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.511437 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a43dcb6-b650-498f-9346-829864c2737a" containerName="payload-pre-processing" Oct 30 12:21:40.511634 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.511574 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ae19ef-b176-41c3-b364-09041b61bd79" containerName="payload-processing" Oct 30 12:21:40.511634 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.511588 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a43dcb6-b650-498f-9346-829864c2737a" containerName="payload-pre-processing" Oct 30 12:21:40.514703 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.514675 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" Oct 30 12:21:40.543436 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.529356 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ebc6bb-serving-cert\"" Oct 30 12:21:40.543436 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.535602 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt"] Oct 30 12:21:40.621450 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.621409 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brdbk\" (UniqueName: \"kubernetes.io/projected/9f251628-d235-4066-9bb0-26067f9356db-kube-api-access-brdbk\") pod \"maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt\" (UID: \"9f251628-d235-4066-9bb0-26067f9356db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" Oct 30 12:21:40.621450 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.621461 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f251628-d235-4066-9bb0-26067f9356db-openshift-service-ca\") pod \"maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt\" (UID: \"9f251628-d235-4066-9bb0-26067f9356db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" Oct 30 12:21:40.621698 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.621668 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f251628-d235-4066-9bb0-26067f9356db-metrics-certs\") pod \"maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt\" (UID: \"9f251628-d235-4066-9bb0-26067f9356db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" Oct 30 12:21:40.621698 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.621694 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f251628-d235-4066-9bb0-26067f9356db-maas-api-tls\") pod \"maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt\" (UID: \"9f251628-d235-4066-9bb0-26067f9356db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" Oct 30 12:21:40.728093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.727230 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brdbk\" (UniqueName: \"kubernetes.io/projected/9f251628-d235-4066-9bb0-26067f9356db-kube-api-access-brdbk\") pod \"maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt\" (UID: \"9f251628-d235-4066-9bb0-26067f9356db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" Oct 30 12:21:40.728093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.727318 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f251628-d235-4066-9bb0-26067f9356db-openshift-service-ca\") pod \"maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt\" (UID: \"9f251628-d235-4066-9bb0-26067f9356db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" Oct 30 12:21:40.728093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.727585 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f251628-d235-4066-9bb0-26067f9356db-metrics-certs\") pod \"maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt\" (UID: \"9f251628-d235-4066-9bb0-26067f9356db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" Oct 30 12:21:40.728093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.727625 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f251628-d235-4066-9bb0-26067f9356db-maas-api-tls\") pod \"maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt\" (UID: \"9f251628-d235-4066-9bb0-26067f9356db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" Oct 30 12:21:40.736949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.736914 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f251628-d235-4066-9bb0-26067f9356db-openshift-service-ca\") pod \"maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt\" (UID: \"9f251628-d235-4066-9bb0-26067f9356db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" Oct 30 12:21:40.739440 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.739355 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f251628-d235-4066-9bb0-26067f9356db-maas-api-tls\") pod \"maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt\" (UID: \"9f251628-d235-4066-9bb0-26067f9356db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" Oct 30 12:21:40.761147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.758600 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f251628-d235-4066-9bb0-26067f9356db-metrics-certs\") pod \"maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt\" (UID: \"9f251628-d235-4066-9bb0-26067f9356db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" Oct 30 12:21:40.794680 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.786838 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brdbk\" (UniqueName: \"kubernetes.io/projected/9f251628-d235-4066-9bb0-26067f9356db-kube-api-access-brdbk\") pod \"maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt\" (UID: \"9f251628-d235-4066-9bb0-26067f9356db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" Oct 30 12:21:40.839663 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:40.838438 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" Oct 30 12:21:41.165175 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:21:41.165139 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f251628_d235_4066_9bb0_26067f9356db.slice/crio-6e8817134bae3b54070985196ec20e1cf5296ce0b7ea469d11b8aa9c134ba14e WatchSource:0}: Error finding container 6e8817134bae3b54070985196ec20e1cf5296ce0b7ea469d11b8aa9c134ba14e: Status 404 returned error can't find the container with id 6e8817134bae3b54070985196ec20e1cf5296ce0b7ea469d11b8aa9c134ba14e Oct 30 12:21:41.166306 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:41.166281 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt"] Oct 30 12:21:41.207659 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:41.207623 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" event={"ID":"9f251628-d235-4066-9bb0-26067f9356db","Type":"ContainerStarted","Data":"6e8817134bae3b54070985196ec20e1cf5296ce0b7ea469d11b8aa9c134ba14e"} Oct 30 12:21:41.682216 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:41.682160 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" podUID="6337aeed-f473-40d3-89eb-ac6f6e8bd7a1" containerName="payload-pre-processing" containerID="cri-o://661849857db2cb462142d2710a40e4a7ff31bcab3cf8a6a49c06450384e8887c" gracePeriod=25 Oct 30 12:21:42.216449 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:42.216406 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" event={"ID":"9f251628-d235-4066-9bb0-26067f9356db","Type":"ContainerStarted","Data":"d83283cb6c1abb2135c8d36ea51346a50f711c3108b8631a812bcecb70e415c6"} Oct 30 12:21:42.216759 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:42.216737 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" Oct 30 12:21:42.258110 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:42.258039 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" podStartSLOduration=1.831817714 podStartE2EDuration="2.258023734s" totalImagesPullingTime="426.20602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:21:42.256749673 +0000 UTC m=+1653.290888069" watchObservedRunningTime="2026-10-30 12:21:42.258023734 +0000 UTC m=+1653.292162129" Oct 30 12:21:43.765905 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:43.765863 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv"] Oct 30 12:21:43.766452 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:43.766148 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" podUID="8a8befff-f988-4749-948f-0f4e0ca2ff1e" containerName="maas-api" containerID="cri-o://24852d19c7e0e03095e5a766f51b059c389a4b8156d91b62f3e36f124d13a78b" gracePeriod=30 Oct 30 12:21:44.392527 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:44.392462 4468 generic.go:356] "Generic (PLEG): container finished" podID="8a8befff-f988-4749-948f-0f4e0ca2ff1e" containerID="24852d19c7e0e03095e5a766f51b059c389a4b8156d91b62f3e36f124d13a78b" exitCode=0 Oct 30 12:21:44.400851 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:44.400811 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" event={"ID":"8a8befff-f988-4749-948f-0f4e0ca2ff1e","Type":"ContainerDied","Data":"24852d19c7e0e03095e5a766f51b059c389a4b8156d91b62f3e36f124d13a78b"} Oct 30 12:21:44.480577 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:44.480538 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" Oct 30 12:21:44.493600 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:44.493566 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a8befff-f988-4749-948f-0f4e0ca2ff1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8befff-f988-4749-948f-0f4e0ca2ff1e-maas-api-tls\") pod \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\" (UID: \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\") " Oct 30 12:21:44.494526 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:44.494312 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a8befff-f988-4749-948f-0f4e0ca2ff1e-kube-api-access-b2nht\" (UniqueName: \"kubernetes.io/projected/8a8befff-f988-4749-948f-0f4e0ca2ff1e-kube-api-access-b2nht\") pod \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\" (UID: \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\") " Oct 30 12:21:44.496009 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:44.495986 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a8befff-f988-4749-948f-0f4e0ca2ff1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8befff-f988-4749-948f-0f4e0ca2ff1e-openshift-service-ca\") pod \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\" (UID: \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\") " Oct 30 12:21:44.496315 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:44.496286 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a8befff-f988-4749-948f-0f4e0ca2ff1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8befff-f988-4749-948f-0f4e0ca2ff1e-metrics-certs\") pod \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\" (UID: \"8a8befff-f988-4749-948f-0f4e0ca2ff1e\") " Oct 30 12:21:44.502214 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:44.497500 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8befff-f988-4749-948f-0f4e0ca2ff1e-openshift-service-ca" pod "8a8befff-f988-4749-948f-0f4e0ca2ff1e" (UID: "8a8befff-f988-4749-948f-0f4e0ca2ff1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:21:44.518091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:44.514742 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8befff-f988-4749-948f-0f4e0ca2ff1e-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:21:44.518278 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:44.518233 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a8befff-f988-4749-948f-0f4e0ca2ff1e-maas-api-tls" pod "8a8befff-f988-4749-948f-0f4e0ca2ff1e" (UID: "8a8befff-f988-4749-948f-0f4e0ca2ff1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:21:44.530090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:44.522230 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a8befff-f988-4749-948f-0f4e0ca2ff1e-metrics-certs" pod "8a8befff-f988-4749-948f-0f4e0ca2ff1e" (UID: "8a8befff-f988-4749-948f-0f4e0ca2ff1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:21:44.530090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:44.529334 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a8befff-f988-4749-948f-0f4e0ca2ff1e-kube-api-access-b2nht" pod "8a8befff-f988-4749-948f-0f4e0ca2ff1e" (UID: "8a8befff-f988-4749-948f-0f4e0ca2ff1e"). InnerVolumeSpecName "kube-api-access-b2nht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:21:44.616251 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:44.616203 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2nht\" (UniqueName: \"kubernetes.io/projected/8a8befff-f988-4749-948f-0f4e0ca2ff1e-kube-api-access-b2nht\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:21:44.616251 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:44.616246 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8befff-f988-4749-948f-0f4e0ca2ff1e-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:21:44.616535 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:44.616262 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8befff-f988-4749-948f-0f4e0ca2ff1e-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:21:45.410726 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:45.410694 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" Oct 30 12:21:45.411189 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:45.410724 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv" event={"ID":"8a8befff-f988-4749-948f-0f4e0ca2ff1e","Type":"ContainerDied","Data":"082ef7f678b36b29be552dd893590085874a7df62bdbcd59f779e6e04e4a5644"} Oct 30 12:21:45.411189 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:45.410792 4468 scope.go:122] "RemoveContainer" containerID="24852d19c7e0e03095e5a766f51b059c389a4b8156d91b62f3e36f124d13a78b" Oct 30 12:21:45.463404 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:45.463365 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv"] Oct 30 12:21:45.474766 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:45.474728 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5649cd-7dd8c8865f-bqdvv"] Oct 30 12:21:47.091672 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:47.091631 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a8befff-f988-4749-948f-0f4e0ca2ff1e" path="/var/lib/kubelet/pods/8a8befff-f988-4749-948f-0f4e0ca2ff1e/volumes" Oct 30 12:21:48.229812 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:48.229775 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" Oct 30 12:21:51.570161 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:51.561253 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" podUID="6337aeed-f473-40d3-89eb-ac6f6e8bd7a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:21:53.640191 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:21:53.640151 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:21:59.478552 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.478512 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl"] Oct 30 12:21:59.479657 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.479635 4468 cpu_manager.go:414] "containerMap: removing container" podUID="8a8befff-f988-4749-948f-0f4e0ca2ff1e" containerName="maas-api" Oct 30 12:21:59.479742 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.479658 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a8befff-f988-4749-948f-0f4e0ca2ff1e" containerName="maas-api" Oct 30 12:21:59.479812 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.479800 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a8befff-f988-4749-948f-0f4e0ca2ff1e" containerName="maas-api" Oct 30 12:21:59.485506 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.485479 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.511952 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.511912 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl"] Oct 30 12:21:59.515044 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.515011 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c35d6b-openshift-default-dockercfg-wdrvr\"" Oct 30 12:21:59.606096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.599708 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-istio-envoy\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.606096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.599772 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9321539-0223-49a1-b3bf-765475cb0057-istiod-ca-cert\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.606096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.599816 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-workload-socket\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.606096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.599961 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9321539-0223-49a1-b3bf-765475cb0057-istio-token\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.606096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.600027 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46mtf\" (UniqueName: \"kubernetes.io/projected/c9321539-0223-49a1-b3bf-765475cb0057-kube-api-access-46mtf\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.606096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.600100 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-credential-socket\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.606096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.600168 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-istio-data\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.606096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.600225 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-workload-certs\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.606096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.600295 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9321539-0223-49a1-b3bf-765475cb0057-istio-podinfo\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.707269 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.707227 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-istio-data\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.739586 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.739492 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-workload-certs\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.758094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.744224 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9321539-0223-49a1-b3bf-765475cb0057-istio-podinfo\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.765637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.765592 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-istio-envoy\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.766639 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.707914 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-istio-data\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.766639 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.740201 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-workload-certs\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.768119 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.767913 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9321539-0223-49a1-b3bf-765475cb0057-istiod-ca-cert\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.768583 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.766445 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9321539-0223-49a1-b3bf-765475cb0057-istiod-ca-cert\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.768770 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.768751 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-workload-socket\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.769295 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.769158 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-workload-socket\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.769633 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.769608 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9321539-0223-49a1-b3bf-765475cb0057-istio-token\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.771290 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.771269 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46mtf\" (UniqueName: \"kubernetes.io/projected/c9321539-0223-49a1-b3bf-765475cb0057-kube-api-access-46mtf\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.771840 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.771820 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-credential-socket\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.772599 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.772334 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-credential-socket\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.820652 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.820614 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46mtf\" (UniqueName: \"kubernetes.io/projected/c9321539-0223-49a1-b3bf-765475cb0057-kube-api-access-46mtf\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.825703 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.825668 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-istio-envoy\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.841480 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.841399 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9321539-0223-49a1-b3bf-765475cb0057-istio-podinfo\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:21:59.849778 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:21:59.849699 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9321539-0223-49a1-b3bf-765475cb0057-istio-token\") pod \"e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:22:00.119157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:00.118802 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:22:00.528437 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:22:00.528403 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9321539_0223_49a1_b3bf_765475cb0057.slice/crio-8e8bb270efc77c4862629a99a04f651cf8e710ddb86ae5a9104cba73b75e343d WatchSource:0}: Error finding container 8e8bb270efc77c4862629a99a04f651cf8e710ddb86ae5a9104cba73b75e343d: Status 404 returned error can't find the container with id 8e8bb270efc77c4862629a99a04f651cf8e710ddb86ae5a9104cba73b75e343d Oct 30 12:22:00.529161 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:00.529059 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl"] Oct 30 12:22:00.531349 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:00.531316 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:22:00.531432 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:00.531389 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:22:00.531432 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:00.531424 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:22:00.771162 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:00.771123 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" event={"ID":"c9321539-0223-49a1-b3bf-765475cb0057","Type":"ContainerStarted","Data":"1cd10abeec7e993d9f973d6e4168a7d8ac1c6fbc2c439269018e6b6642b40c9d"} Oct 30 12:22:00.771340 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:00.771191 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" event={"ID":"c9321539-0223-49a1-b3bf-765475cb0057","Type":"ContainerStarted","Data":"8e8bb270efc77c4862629a99a04f651cf8e710ddb86ae5a9104cba73b75e343d"} Oct 30 12:22:00.816775 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:00.816662 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" podStartSLOduration=1.8166467769999999 podStartE2EDuration="1.816646777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:00.814573808 +0000 UTC m=+1671.848712205" watchObservedRunningTime="2026-10-30 12:22:00.816646777 +0000 UTC m=+1671.850785173" Oct 30 12:22:01.123739 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:01.123651 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:22:01.164976 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:01.163899 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:22:01.534769 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:01.534727 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" podUID="6337aeed-f473-40d3-89eb-ac6f6e8bd7a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:22:01.780242 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:01.780201 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:22:01.781404 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:01.781381 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:22:02.033895 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:02.033866 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" Oct 30 12:22:02.154690 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:02.154597 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6337aeed-f473-40d3-89eb-ac6f6e8bd7a1-kube-api-access-fwhb2\" (UniqueName: \"kubernetes.io/projected/6337aeed-f473-40d3-89eb-ac6f6e8bd7a1-kube-api-access-fwhb2\") pod \"6337aeed-f473-40d3-89eb-ac6f6e8bd7a1\" (UID: \"6337aeed-f473-40d3-89eb-ac6f6e8bd7a1\") " Oct 30 12:22:02.154848 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:02.154712 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6337aeed-f473-40d3-89eb-ac6f6e8bd7a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6337aeed-f473-40d3-89eb-ac6f6e8bd7a1-plugins-config-volume\") pod \"6337aeed-f473-40d3-89eb-ac6f6e8bd7a1\" (UID: \"6337aeed-f473-40d3-89eb-ac6f6e8bd7a1\") " Oct 30 12:22:02.155120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:02.155081 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6337aeed-f473-40d3-89eb-ac6f6e8bd7a1-plugins-config-volume" pod "6337aeed-f473-40d3-89eb-ac6f6e8bd7a1" (UID: "6337aeed-f473-40d3-89eb-ac6f6e8bd7a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:22:02.155292 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:02.155274 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6337aeed-f473-40d3-89eb-ac6f6e8bd7a1-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:02.156850 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:02.156827 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6337aeed-f473-40d3-89eb-ac6f6e8bd7a1-kube-api-access-fwhb2" pod "6337aeed-f473-40d3-89eb-ac6f6e8bd7a1" (UID: "6337aeed-f473-40d3-89eb-ac6f6e8bd7a1"). InnerVolumeSpecName "kube-api-access-fwhb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:22:02.256764 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:02.256698 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwhb2\" (UniqueName: \"kubernetes.io/projected/6337aeed-f473-40d3-89eb-ac6f6e8bd7a1-kube-api-access-fwhb2\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:02.788200 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:02.788156 4468 generic.go:356] "Generic (PLEG): container finished" podID="6337aeed-f473-40d3-89eb-ac6f6e8bd7a1" containerID="661849857db2cb462142d2710a40e4a7ff31bcab3cf8a6a49c06450384e8887c" exitCode=0 Oct 30 12:22:02.788701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:02.788222 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" Oct 30 12:22:02.788701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:02.788246 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" event={"ID":"6337aeed-f473-40d3-89eb-ac6f6e8bd7a1","Type":"ContainerDied","Data":"661849857db2cb462142d2710a40e4a7ff31bcab3cf8a6a49c06450384e8887c"} Oct 30 12:22:02.788701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:02.788302 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g" event={"ID":"6337aeed-f473-40d3-89eb-ac6f6e8bd7a1","Type":"ContainerDied","Data":"6c070e7abfb7841ee068aec183102bd1df4e8317fc0f2855f97ee2362f2a438b"} Oct 30 12:22:02.788701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:02.788331 4468 scope.go:122] "RemoveContainer" containerID="661849857db2cb462142d2710a40e4a7ff31bcab3cf8a6a49c06450384e8887c" Oct 30 12:22:02.837295 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:02.837247 4468 scope.go:122] "RemoveContainer" containerID="661849857db2cb462142d2710a40e4a7ff31bcab3cf8a6a49c06450384e8887c" Oct 30 12:22:02.838145 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:22:02.838000 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661849857db2cb462142d2710a40e4a7ff31bcab3cf8a6a49c06450384e8887c\": container with ID starting with 661849857db2cb462142d2710a40e4a7ff31bcab3cf8a6a49c06450384e8887c not found: ID does not exist" containerID="661849857db2cb462142d2710a40e4a7ff31bcab3cf8a6a49c06450384e8887c" Oct 30 12:22:02.838145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:02.838046 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661849857db2cb462142d2710a40e4a7ff31bcab3cf8a6a49c06450384e8887c"} err="failed to get container status \"661849857db2cb462142d2710a40e4a7ff31bcab3cf8a6a49c06450384e8887c\": rpc error: code = NotFound desc = could not find container \"661849857db2cb462142d2710a40e4a7ff31bcab3cf8a6a49c06450384e8887c\": container with ID starting with 661849857db2cb462142d2710a40e4a7ff31bcab3cf8a6a49c06450384e8887c not found: ID does not exist" Oct 30 12:22:02.908963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:02.908926 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g"] Oct 30 12:22:02.928135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:02.922463 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-tmq4g"] Oct 30 12:22:03.001343 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:03.001304 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8"] Oct 30 12:22:03.097830 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:03.097734 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6337aeed-f473-40d3-89eb-ac6f6e8bd7a1" path="/var/lib/kubelet/pods/6337aeed-f473-40d3-89eb-ac6f6e8bd7a1/volumes" Oct 30 12:22:06.879867 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:06.856363 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk"] Oct 30 12:22:06.879867 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:06.879420 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" podUID="5593fe4c-beb7-4fa5-92b1-e023a25e186c" containerName="maas-api" containerID="cri-o://1c5b1fc7289db493cbd8b137a37631dec8675a44ac983ba00be4e78069646e3d" gracePeriod=30 Oct 30 12:22:07.730597 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:07.730564 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" Oct 30 12:22:07.939195 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:07.938568 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5593fe4c-beb7-4fa5-92b1-e023a25e186c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5593fe4c-beb7-4fa5-92b1-e023a25e186c-metrics-certs\") pod \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\" (UID: \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\") " Oct 30 12:22:07.939195 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:07.938660 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5593fe4c-beb7-4fa5-92b1-e023a25e186c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5593fe4c-beb7-4fa5-92b1-e023a25e186c-maas-api-tls\") pod \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\" (UID: \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\") " Oct 30 12:22:07.939195 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:07.938714 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5593fe4c-beb7-4fa5-92b1-e023a25e186c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5593fe4c-beb7-4fa5-92b1-e023a25e186c-openshift-service-ca\") pod \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\" (UID: \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\") " Oct 30 12:22:07.939195 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:07.938836 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5593fe4c-beb7-4fa5-92b1-e023a25e186c-kube-api-access-sg8d7\" (UniqueName: \"kubernetes.io/projected/5593fe4c-beb7-4fa5-92b1-e023a25e186c-kube-api-access-sg8d7\") pod \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\" (UID: \"5593fe4c-beb7-4fa5-92b1-e023a25e186c\") " Oct 30 12:22:07.983640 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:07.983595 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5593fe4c-beb7-4fa5-92b1-e023a25e186c-maas-api-tls" pod "5593fe4c-beb7-4fa5-92b1-e023a25e186c" (UID: "5593fe4c-beb7-4fa5-92b1-e023a25e186c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:22:07.998466 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:07.998416 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5593fe4c-beb7-4fa5-92b1-e023a25e186c-openshift-service-ca" pod "5593fe4c-beb7-4fa5-92b1-e023a25e186c" (UID: "5593fe4c-beb7-4fa5-92b1-e023a25e186c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:22:08.003646 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.003548 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8" podUID="61651192-0a38-49aa-8b90-20a364a9761b" containerName="payload-processing" containerID="cri-o://2126ec66266f0f11373a60d504bb51107ff580f41fb0dbae22885ab3883db769" gracePeriod=25 Oct 30 12:22:08.020154 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.020113 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5593fe4c-beb7-4fa5-92b1-e023a25e186c-metrics-certs" pod "5593fe4c-beb7-4fa5-92b1-e023a25e186c" (UID: "5593fe4c-beb7-4fa5-92b1-e023a25e186c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:22:08.025966 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.025927 4468 generic.go:356] "Generic (PLEG): container finished" podID="5593fe4c-beb7-4fa5-92b1-e023a25e186c" containerID="1c5b1fc7289db493cbd8b137a37631dec8675a44ac983ba00be4e78069646e3d" exitCode=0 Oct 30 12:22:08.026144 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.026123 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" event={"ID":"5593fe4c-beb7-4fa5-92b1-e023a25e186c","Type":"ContainerDied","Data":"1c5b1fc7289db493cbd8b137a37631dec8675a44ac983ba00be4e78069646e3d"} Oct 30 12:22:08.026240 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.026226 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" event={"ID":"5593fe4c-beb7-4fa5-92b1-e023a25e186c","Type":"ContainerDied","Data":"bfbce2e21f8ad70fe9f594e3b91bfa70502b7659e38b5d51a1925442fef1ee77"} Oct 30 12:22:08.026280 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.026267 4468 scope.go:122] "RemoveContainer" containerID="1c5b1fc7289db493cbd8b137a37631dec8675a44ac983ba00be4e78069646e3d" Oct 30 12:22:08.026502 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.026485 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk" Oct 30 12:22:08.029802 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.029767 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5593fe4c-beb7-4fa5-92b1-e023a25e186c-kube-api-access-sg8d7" pod "5593fe4c-beb7-4fa5-92b1-e023a25e186c" (UID: "5593fe4c-beb7-4fa5-92b1-e023a25e186c"). InnerVolumeSpecName "kube-api-access-sg8d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:22:08.053084 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.053035 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg8d7\" (UniqueName: \"kubernetes.io/projected/5593fe4c-beb7-4fa5-92b1-e023a25e186c-kube-api-access-sg8d7\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:08.053213 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.053091 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5593fe4c-beb7-4fa5-92b1-e023a25e186c-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:08.053213 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.053106 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5593fe4c-beb7-4fa5-92b1-e023a25e186c-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:08.053213 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.053118 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5593fe4c-beb7-4fa5-92b1-e023a25e186c-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:08.135320 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.135289 4468 scope.go:122] "RemoveContainer" containerID="1c5b1fc7289db493cbd8b137a37631dec8675a44ac983ba00be4e78069646e3d" Oct 30 12:22:08.135941 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:22:08.135914 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c5b1fc7289db493cbd8b137a37631dec8675a44ac983ba00be4e78069646e3d\": container with ID starting with 1c5b1fc7289db493cbd8b137a37631dec8675a44ac983ba00be4e78069646e3d not found: ID does not exist" containerID="1c5b1fc7289db493cbd8b137a37631dec8675a44ac983ba00be4e78069646e3d" Oct 30 12:22:08.136059 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.135957 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5b1fc7289db493cbd8b137a37631dec8675a44ac983ba00be4e78069646e3d"} err="failed to get container status \"1c5b1fc7289db493cbd8b137a37631dec8675a44ac983ba00be4e78069646e3d\": rpc error: code = NotFound desc = could not find container \"1c5b1fc7289db493cbd8b137a37631dec8675a44ac983ba00be4e78069646e3d\": container with ID starting with 1c5b1fc7289db493cbd8b137a37631dec8675a44ac983ba00be4e78069646e3d not found: ID does not exist" Oct 30 12:22:08.324040 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.324011 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8" Oct 30 12:22:08.359294 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.359252 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61651192-0a38-49aa-8b90-20a364a9761b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61651192-0a38-49aa-8b90-20a364a9761b-plugins-config-volume\") pod \"61651192-0a38-49aa-8b90-20a364a9761b\" (UID: \"61651192-0a38-49aa-8b90-20a364a9761b\") " Oct 30 12:22:08.359532 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.359518 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61651192-0a38-49aa-8b90-20a364a9761b-kube-api-access-b6pxs\" (UniqueName: \"kubernetes.io/projected/61651192-0a38-49aa-8b90-20a364a9761b-kube-api-access-b6pxs\") pod \"61651192-0a38-49aa-8b90-20a364a9761b\" (UID: \"61651192-0a38-49aa-8b90-20a364a9761b\") " Oct 30 12:22:08.359726 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.359699 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61651192-0a38-49aa-8b90-20a364a9761b-plugins-config-volume" pod "61651192-0a38-49aa-8b90-20a364a9761b" (UID: "61651192-0a38-49aa-8b90-20a364a9761b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:22:08.360097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.360055 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61651192-0a38-49aa-8b90-20a364a9761b-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:08.362349 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.362309 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61651192-0a38-49aa-8b90-20a364a9761b-kube-api-access-b6pxs" pod "61651192-0a38-49aa-8b90-20a364a9761b" (UID: "61651192-0a38-49aa-8b90-20a364a9761b"). InnerVolumeSpecName "kube-api-access-b6pxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:22:08.399868 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.399830 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk"] Oct 30 12:22:08.404130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.404092 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-qqfwk"] Oct 30 12:22:08.461873 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:08.461828 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6pxs\" (UniqueName: \"kubernetes.io/projected/61651192-0a38-49aa-8b90-20a364a9761b-kube-api-access-b6pxs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:09.036205 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:09.036167 4468 generic.go:356] "Generic (PLEG): container finished" podID="61651192-0a38-49aa-8b90-20a364a9761b" containerID="2126ec66266f0f11373a60d504bb51107ff580f41fb0dbae22885ab3883db769" exitCode=0 Oct 30 12:22:09.036652 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:09.036210 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8" Oct 30 12:22:09.036652 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:09.036249 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8" event={"ID":"61651192-0a38-49aa-8b90-20a364a9761b","Type":"ContainerDied","Data":"2126ec66266f0f11373a60d504bb51107ff580f41fb0dbae22885ab3883db769"} Oct 30 12:22:09.036652 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:09.036297 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8" event={"ID":"61651192-0a38-49aa-8b90-20a364a9761b","Type":"ContainerDied","Data":"e673b74a24a09617604e272321bf463ff197ca37fbf294b6b097c9694a16bded"} Oct 30 12:22:09.036652 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:09.036320 4468 scope.go:122] "RemoveContainer" containerID="2126ec66266f0f11373a60d504bb51107ff580f41fb0dbae22885ab3883db769" Oct 30 12:22:09.047222 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:09.047187 4468 scope.go:122] "RemoveContainer" containerID="2126ec66266f0f11373a60d504bb51107ff580f41fb0dbae22885ab3883db769" Oct 30 12:22:09.047556 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:22:09.047535 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2126ec66266f0f11373a60d504bb51107ff580f41fb0dbae22885ab3883db769\": container with ID starting with 2126ec66266f0f11373a60d504bb51107ff580f41fb0dbae22885ab3883db769 not found: ID does not exist" containerID="2126ec66266f0f11373a60d504bb51107ff580f41fb0dbae22885ab3883db769" Oct 30 12:22:09.047609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:09.047567 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2126ec66266f0f11373a60d504bb51107ff580f41fb0dbae22885ab3883db769"} err="failed to get container status \"2126ec66266f0f11373a60d504bb51107ff580f41fb0dbae22885ab3883db769\": rpc error: code = NotFound desc = could not find container \"2126ec66266f0f11373a60d504bb51107ff580f41fb0dbae22885ab3883db769\": container with ID starting with 2126ec66266f0f11373a60d504bb51107ff580f41fb0dbae22885ab3883db769 not found: ID does not exist" Oct 30 12:22:09.093968 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:09.093930 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5593fe4c-beb7-4fa5-92b1-e023a25e186c" path="/var/lib/kubelet/pods/5593fe4c-beb7-4fa5-92b1-e023a25e186c/volumes" Oct 30 12:22:09.094309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:09.094294 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8"] Oct 30 12:22:09.094351 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:09.094331 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7566f5-bdbbcf4bb-795v8"] Oct 30 12:22:11.091610 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:11.091574 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61651192-0a38-49aa-8b90-20a364a9761b" path="/var/lib/kubelet/pods/61651192-0a38-49aa-8b90-20a364a9761b/volumes" Oct 30 12:22:15.782005 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:15.781954 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8"] Oct 30 12:22:15.848096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:15.847151 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb"] Oct 30 12:22:19.135180 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.135140 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq"] Oct 30 12:22:19.135811 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.135797 4468 cpu_manager.go:414] "containerMap: removing container" podUID="5593fe4c-beb7-4fa5-92b1-e023a25e186c" containerName="maas-api" Oct 30 12:22:19.135853 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.135811 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5593fe4c-beb7-4fa5-92b1-e023a25e186c" containerName="maas-api" Oct 30 12:22:19.135853 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.135827 4468 cpu_manager.go:414] "containerMap: removing container" podUID="61651192-0a38-49aa-8b90-20a364a9761b" containerName="payload-processing" Oct 30 12:22:19.135853 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.135833 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61651192-0a38-49aa-8b90-20a364a9761b" containerName="payload-processing" Oct 30 12:22:19.135951 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.135872 4468 cpu_manager.go:414] "containerMap: removing container" podUID="6337aeed-f473-40d3-89eb-ac6f6e8bd7a1" containerName="payload-pre-processing" Oct 30 12:22:19.135951 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.135880 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6337aeed-f473-40d3-89eb-ac6f6e8bd7a1" containerName="payload-pre-processing" Oct 30 12:22:19.136015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.135974 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6337aeed-f473-40d3-89eb-ac6f6e8bd7a1" containerName="payload-pre-processing" Oct 30 12:22:19.136015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.135986 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="5593fe4c-beb7-4fa5-92b1-e023a25e186c" containerName="maas-api" Oct 30 12:22:19.136015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.135991 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="61651192-0a38-49aa-8b90-20a364a9761b" containerName="payload-processing" Oct 30 12:22:19.141744 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.141719 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" Oct 30 12:22:19.158182 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.158150 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq"] Oct 30 12:22:19.165500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.165461 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c35d6b-serving-cert\"" Oct 30 12:22:19.274503 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.274465 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-openshift-service-ca\") pod \"maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq\" (UID: \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" Oct 30 12:22:19.274702 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.274641 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kwdj\" (UniqueName: \"kubernetes.io/projected/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-kube-api-access-5kwdj\") pod \"maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq\" (UID: \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" Oct 30 12:22:19.274795 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.274783 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-maas-api-tls\") pod \"maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq\" (UID: \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" Oct 30 12:22:19.274845 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.274835 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-metrics-certs\") pod \"maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq\" (UID: \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" Oct 30 12:22:19.380092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.379657 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kwdj\" (UniqueName: \"kubernetes.io/projected/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-kube-api-access-5kwdj\") pod \"maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq\" (UID: \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" Oct 30 12:22:19.380092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.379863 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-maas-api-tls\") pod \"maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq\" (UID: \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" Oct 30 12:22:19.380092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.379947 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-metrics-certs\") pod \"maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq\" (UID: \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" Oct 30 12:22:19.380092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.380088 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-openshift-service-ca\") pod \"maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq\" (UID: \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" Oct 30 12:22:19.381246 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.381093 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-openshift-service-ca\") pod \"maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq\" (UID: \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" Oct 30 12:22:19.420787 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.420715 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kwdj\" (UniqueName: \"kubernetes.io/projected/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-kube-api-access-5kwdj\") pod \"maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq\" (UID: \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" Oct 30 12:22:19.425736 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.425702 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-metrics-certs\") pod \"maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq\" (UID: \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" Oct 30 12:22:19.428283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.426661 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-maas-api-tls\") pod \"maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq\" (UID: \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" Oct 30 12:22:19.458741 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.458707 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" Oct 30 12:22:19.835795 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:22:19.835760 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7f730a_2bc9_4f4a_8bb6_bedc1a59bc37.slice/crio-1ef90e4a5644c22f8c03a7348e0874e6418bec8ff9b2609144c0b9a1b097d5a1 WatchSource:0}: Error finding container 1ef90e4a5644c22f8c03a7348e0874e6418bec8ff9b2609144c0b9a1b097d5a1: Status 404 returned error can't find the container with id 1ef90e4a5644c22f8c03a7348e0874e6418bec8ff9b2609144c0b9a1b097d5a1 Oct 30 12:22:19.835959 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:19.835856 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq"] Oct 30 12:22:20.230553 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:20.230514 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" event={"ID":"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37","Type":"ContainerStarted","Data":"1ef90e4a5644c22f8c03a7348e0874e6418bec8ff9b2609144c0b9a1b097d5a1"} Oct 30 12:22:20.427660 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:20.427614 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf"] Oct 30 12:22:20.432137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:20.432107 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf" Oct 30 12:22:20.452313 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:20.452281 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf"] Oct 30 12:22:20.457595 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:20.457509 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-060f41f2\"" Oct 30 12:22:20.462893 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:20.462854 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-060f41f2-dockercfg-fl5vp\"" Oct 30 12:22:20.833819 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:20.777977 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d961b4b0-b85f-4c77-afaa-3ae13e12ed30-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf\" (UID: \"d961b4b0-b85f-4c77-afaa-3ae13e12ed30\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf" Oct 30 12:22:20.833819 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:20.778063 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhtzv\" (UniqueName: \"kubernetes.io/projected/d961b4b0-b85f-4c77-afaa-3ae13e12ed30-kube-api-access-vhtzv\") pod \"payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf\" (UID: \"d961b4b0-b85f-4c77-afaa-3ae13e12ed30\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf" Oct 30 12:22:20.833819 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:20.784167 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8" podUID="82d42735-89ea-4146-853e-2e3382b4547a" containerName="payload-processing" containerID="cri-o://9f32ed83bb00095fdff6ed4bfe82e85eee31aaf5fb7804e5d3e1a680a169b147" gracePeriod=25 Oct 30 12:22:20.875336 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:20.867862 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw"] Oct 30 12:22:20.875336 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:20.868546 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb" podUID="e384fc75-3e1d-466d-8794-6e43f37e6e22" containerName="payload-pre-processing" containerID="cri-o://040f3c54dec2d4aa3dc1e135074a03ff66b4febe774c7b5572ce18c700cb3253" gracePeriod=25 Oct 30 12:22:20.906107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:20.890745 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw" Oct 30 12:22:20.906107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:20.897342 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw"] Oct 30 12:22:20.906107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:20.901854 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbhdl\" (UniqueName: \"kubernetes.io/projected/cb9b76ae-a152-44b0-b00c-4111be5a652d-kube-api-access-nbhdl\") pod \"payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw\" (UID: \"cb9b76ae-a152-44b0-b00c-4111be5a652d\") " pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw" Oct 30 12:22:20.906107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:20.901934 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d961b4b0-b85f-4c77-afaa-3ae13e12ed30-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf\" (UID: \"d961b4b0-b85f-4c77-afaa-3ae13e12ed30\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf" Oct 30 12:22:20.906107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:20.901993 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhtzv\" (UniqueName: \"kubernetes.io/projected/d961b4b0-b85f-4c77-afaa-3ae13e12ed30-kube-api-access-vhtzv\") pod \"payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf\" (UID: \"d961b4b0-b85f-4c77-afaa-3ae13e12ed30\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf" Oct 30 12:22:20.906107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:20.902210 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9b76ae-a152-44b0-b00c-4111be5a652d-plugins-config-volume\") pod \"payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw\" (UID: \"cb9b76ae-a152-44b0-b00c-4111be5a652d\") " pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw" Oct 30 12:22:20.906107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:20.903243 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d961b4b0-b85f-4c77-afaa-3ae13e12ed30-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf\" (UID: \"d961b4b0-b85f-4c77-afaa-3ae13e12ed30\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf" Oct 30 12:22:20.959133 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:20.959044 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhtzv\" (UniqueName: \"kubernetes.io/projected/d961b4b0-b85f-4c77-afaa-3ae13e12ed30-kube-api-access-vhtzv\") pod \"payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf\" (UID: \"d961b4b0-b85f-4c77-afaa-3ae13e12ed30\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf" Oct 30 12:22:20.967895 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:22:20.963051 4468 logging.go:55] [core] [Channel #1961 SubChannel #1962]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.106:50052", ServerName: "10.134.0.106:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.106:50052: connect: connection refused" Oct 30 12:22:20.967895 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:22:20.963211 4468 logging.go:55] [core] [Channel #1963 SubChannel #1964]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.105:50052", ServerName: "10.134.0.105:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.105:50052: connect: connection refused" Oct 30 12:22:21.003670 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.003629 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbhdl\" (UniqueName: \"kubernetes.io/projected/cb9b76ae-a152-44b0-b00c-4111be5a652d-kube-api-access-nbhdl\") pod \"payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw\" (UID: \"cb9b76ae-a152-44b0-b00c-4111be5a652d\") " pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw" Oct 30 12:22:21.003867 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.003830 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9b76ae-a152-44b0-b00c-4111be5a652d-plugins-config-volume\") pod \"payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw\" (UID: \"cb9b76ae-a152-44b0-b00c-4111be5a652d\") " pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw" Oct 30 12:22:21.004934 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.004906 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9b76ae-a152-44b0-b00c-4111be5a652d-plugins-config-volume\") pod \"payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw\" (UID: \"cb9b76ae-a152-44b0-b00c-4111be5a652d\") " pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw" Oct 30 12:22:21.026622 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.026581 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbhdl\" (UniqueName: \"kubernetes.io/projected/cb9b76ae-a152-44b0-b00c-4111be5a652d-kube-api-access-nbhdl\") pod \"payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw\" (UID: \"cb9b76ae-a152-44b0-b00c-4111be5a652d\") " pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw" Oct 30 12:22:21.052534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.052418 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf" Oct 30 12:22:21.069625 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.069434 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw" Oct 30 12:22:21.240917 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.240881 4468 generic.go:356] "Generic (PLEG): container finished" podID="e384fc75-3e1d-466d-8794-6e43f37e6e22" containerID="040f3c54dec2d4aa3dc1e135074a03ff66b4febe774c7b5572ce18c700cb3253" exitCode=0 Oct 30 12:22:21.241448 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.240943 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb" event={"ID":"e384fc75-3e1d-466d-8794-6e43f37e6e22","Type":"ContainerDied","Data":"040f3c54dec2d4aa3dc1e135074a03ff66b4febe774c7b5572ce18c700cb3253"} Oct 30 12:22:21.242877 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.242849 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" event={"ID":"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37","Type":"ContainerStarted","Data":"930f14db1b7488531704a1ac4d62e71e964ff694bddf18d53017189cb394af02"} Oct 30 12:22:21.243047 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.243006 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" Oct 30 12:22:21.244206 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.244112 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb" Oct 30 12:22:21.244796 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.244763 4468 generic.go:356] "Generic (PLEG): container finished" podID="82d42735-89ea-4146-853e-2e3382b4547a" containerID="9f32ed83bb00095fdff6ed4bfe82e85eee31aaf5fb7804e5d3e1a680a169b147" exitCode=0 Oct 30 12:22:21.244912 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.244864 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8" event={"ID":"82d42735-89ea-4146-853e-2e3382b4547a","Type":"ContainerDied","Data":"9f32ed83bb00095fdff6ed4bfe82e85eee31aaf5fb7804e5d3e1a680a169b147"} Oct 30 12:22:21.300111 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.292719 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" podStartSLOduration=1.9236456180000001 podStartE2EDuration="2.292693044s" totalImagesPullingTime="369.047426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:21.289509161 +0000 UTC m=+1692.323647558" watchObservedRunningTime="2026-10-30 12:22:21.292693044 +0000 UTC m=+1692.326831445" Oct 30 12:22:21.308361 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.308323 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e384fc75-3e1d-466d-8794-6e43f37e6e22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e384fc75-3e1d-466d-8794-6e43f37e6e22-plugins-config-volume\") pod \"e384fc75-3e1d-466d-8794-6e43f37e6e22\" (UID: \"e384fc75-3e1d-466d-8794-6e43f37e6e22\") " Oct 30 12:22:21.308565 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.308509 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e384fc75-3e1d-466d-8794-6e43f37e6e22-kube-api-access-hxd6j\" (UniqueName: \"kubernetes.io/projected/e384fc75-3e1d-466d-8794-6e43f37e6e22-kube-api-access-hxd6j\") pod \"e384fc75-3e1d-466d-8794-6e43f37e6e22\" (UID: \"e384fc75-3e1d-466d-8794-6e43f37e6e22\") " Oct 30 12:22:21.308978 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.308944 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e384fc75-3e1d-466d-8794-6e43f37e6e22-plugins-config-volume" pod "e384fc75-3e1d-466d-8794-6e43f37e6e22" (UID: "e384fc75-3e1d-466d-8794-6e43f37e6e22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:22:21.309127 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.309106 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e384fc75-3e1d-466d-8794-6e43f37e6e22-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:21.313365 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.313331 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e384fc75-3e1d-466d-8794-6e43f37e6e22-kube-api-access-hxd6j" pod "e384fc75-3e1d-466d-8794-6e43f37e6e22" (UID: "e384fc75-3e1d-466d-8794-6e43f37e6e22"). InnerVolumeSpecName "kube-api-access-hxd6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:22:21.349564 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.349528 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8" Oct 30 12:22:21.399607 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.398989 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf"] Oct 30 12:22:21.469150 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.447486 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82d42735-89ea-4146-853e-2e3382b4547a-kube-api-access-s874z\" (UniqueName: \"kubernetes.io/projected/82d42735-89ea-4146-853e-2e3382b4547a-kube-api-access-s874z\") pod \"82d42735-89ea-4146-853e-2e3382b4547a\" (UID: \"82d42735-89ea-4146-853e-2e3382b4547a\") " Oct 30 12:22:21.469150 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.447643 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82d42735-89ea-4146-853e-2e3382b4547a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d42735-89ea-4146-853e-2e3382b4547a-plugins-config-volume\") pod \"82d42735-89ea-4146-853e-2e3382b4547a\" (UID: \"82d42735-89ea-4146-853e-2e3382b4547a\") " Oct 30 12:22:21.469150 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.448484 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxd6j\" (UniqueName: \"kubernetes.io/projected/e384fc75-3e1d-466d-8794-6e43f37e6e22-kube-api-access-hxd6j\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:21.469150 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.448930 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d42735-89ea-4146-853e-2e3382b4547a-plugins-config-volume" pod "82d42735-89ea-4146-853e-2e3382b4547a" (UID: "82d42735-89ea-4146-853e-2e3382b4547a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:22:21.474386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.473409 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d42735-89ea-4146-853e-2e3382b4547a-kube-api-access-s874z" pod "82d42735-89ea-4146-853e-2e3382b4547a" (UID: "82d42735-89ea-4146-853e-2e3382b4547a"). InnerVolumeSpecName "kube-api-access-s874z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:22:21.535843 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.529475 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw"] Oct 30 12:22:21.550721 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.550672 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s874z\" (UniqueName: \"kubernetes.io/projected/82d42735-89ea-4146-853e-2e3382b4547a-kube-api-access-s874z\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:21.556165 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.556125 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d42735-89ea-4146-853e-2e3382b4547a-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:21.962094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.962019 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8" podUID="82d42735-89ea-4146-853e-2e3382b4547a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.106:50052\" within 1s: context deadline exceeded" Oct 30 12:22:21.962310 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:21.962136 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb" podUID="e384fc75-3e1d-466d-8794-6e43f37e6e22" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.105:50052\" within 1s: context deadline exceeded" Oct 30 12:22:22.254620 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:22.254509 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf" event={"ID":"d961b4b0-b85f-4c77-afaa-3ae13e12ed30","Type":"ContainerStarted","Data":"80fcda871bed73ed4b068c38bf428826e820c8a2e83d3b8b65c558d2d1cc24e3"} Oct 30 12:22:22.254620 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:22.254578 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf" event={"ID":"d961b4b0-b85f-4c77-afaa-3ae13e12ed30","Type":"ContainerStarted","Data":"80bcf20326ce329f50269338ba7d5d7fac44758aee327e9ea6cdda4dd34beb49"} Oct 30 12:22:22.254620 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:22.254613 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf" Oct 30 12:22:22.256433 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:22.256054 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb" Oct 30 12:22:22.256433 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:22.256081 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb" event={"ID":"e384fc75-3e1d-466d-8794-6e43f37e6e22","Type":"ContainerDied","Data":"8dd61f9e385b20daea45020c77d0b95533358457e47bf64f166d62ada7530152"} Oct 30 12:22:22.256433 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:22.256152 4468 scope.go:122] "RemoveContainer" containerID="040f3c54dec2d4aa3dc1e135074a03ff66b4febe774c7b5572ce18c700cb3253" Oct 30 12:22:22.258054 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:22.258026 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw" event={"ID":"cb9b76ae-a152-44b0-b00c-4111be5a652d","Type":"ContainerStarted","Data":"f3ca78583f3a1f0bece5543ebd7a3d5ea6f928bcabd24d320a5fa38586a7eed5"} Oct 30 12:22:22.258204 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:22.258095 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw" event={"ID":"cb9b76ae-a152-44b0-b00c-4111be5a652d","Type":"ContainerStarted","Data":"cd7cbc6de79828d9ec8fd48a5fbe983c7d42a823a986b933cc1c7781b215fce3"} Oct 30 12:22:22.258204 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:22.258152 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw" Oct 30 12:22:22.259720 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:22.259689 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8" Oct 30 12:22:22.259837 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:22.259711 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8" event={"ID":"82d42735-89ea-4146-853e-2e3382b4547a","Type":"ContainerDied","Data":"71f37c71c0d290d45b248977e586ca5b0d06ced6f4f14a428bceb562b2312728"} Oct 30 12:22:22.267644 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:22.267619 4468 scope.go:122] "RemoveContainer" containerID="9f32ed83bb00095fdff6ed4bfe82e85eee31aaf5fb7804e5d3e1a680a169b147" Oct 30 12:22:22.293707 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:22.293634 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf" podStartSLOduration=2.293614185 podStartE2EDuration="2.293614185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:22.293357579 +0000 UTC m=+1693.327495974" watchObservedRunningTime="2026-10-30 12:22:22.293614185 +0000 UTC m=+1693.327752581" Oct 30 12:22:22.331394 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:22.331358 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb"] Oct 30 12:22:22.335397 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:22.335365 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-82299a-6944cdb7cf-zjdcb"] Oct 30 12:22:22.371811 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:22.371755 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw" podStartSLOduration=2.371731978 podStartE2EDuration="2.371731978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:22.368035071 +0000 UTC m=+1693.402173466" watchObservedRunningTime="2026-10-30 12:22:22.371731978 +0000 UTC m=+1693.405870373" Oct 30 12:22:22.407137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:22.407101 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8"] Oct 30 12:22:22.411317 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:22.411286 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-82299a-765c7fdfd4-kzpr8"] Oct 30 12:22:23.098438 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:23.098395 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82d42735-89ea-4146-853e-2e3382b4547a" path="/var/lib/kubelet/pods/82d42735-89ea-4146-853e-2e3382b4547a/volumes" Oct 30 12:22:23.098913 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:23.098898 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e384fc75-3e1d-466d-8794-6e43f37e6e22" path="/var/lib/kubelet/pods/e384fc75-3e1d-466d-8794-6e43f37e6e22/volumes" Oct 30 12:22:23.600774 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:22:23.600737 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:22:27.266558 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:27.266523 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" Oct 30 12:22:33.301967 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:33.301934 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw" Oct 30 12:22:33.302489 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:33.302013 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf" Oct 30 12:22:41.464742 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:41.464704 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9"] Oct 30 12:22:41.465532 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:41.465502 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" podUID="8278ddb9-839e-4d62-b0b0-ba9cf036ae10" containerName="istio-proxy" containerID="cri-o://987e3f9fb2335a367e6605b2132ae3e5c58c93117883c0b6391a962ff1e5719e" gracePeriod=30 Oct 30 12:22:45.685392 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:45.685345 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4"] Oct 30 12:22:45.686483 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:45.686458 4468 cpu_manager.go:414] "containerMap: removing container" podUID="e384fc75-3e1d-466d-8794-6e43f37e6e22" containerName="payload-pre-processing" Oct 30 12:22:45.686561 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:45.686487 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e384fc75-3e1d-466d-8794-6e43f37e6e22" containerName="payload-pre-processing" Oct 30 12:22:45.686616 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:45.686564 4468 cpu_manager.go:414] "containerMap: removing container" podUID="82d42735-89ea-4146-853e-2e3382b4547a" containerName="payload-processing" Oct 30 12:22:45.686616 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:45.686573 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82d42735-89ea-4146-853e-2e3382b4547a" containerName="payload-processing" Oct 30 12:22:45.686743 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:45.686730 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="82d42735-89ea-4146-853e-2e3382b4547a" containerName="payload-processing" Oct 30 12:22:45.686775 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:45.686755 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="e384fc75-3e1d-466d-8794-6e43f37e6e22" containerName="payload-pre-processing" Oct 30 12:22:45.692146 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:45.692110 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4" Oct 30 12:22:45.706550 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:45.706508 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4"] Oct 30 12:22:45.706831 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:45.706815 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8f2898\"" Oct 30 12:22:45.706902 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:45.706879 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8f2898-dockercfg-bfpc7\"" Oct 30 12:22:45.936333 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:45.935545 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a14ade7a-fdde-453c-8596-e5f2ee3fbba4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4\" (UID: \"a14ade7a-fdde-453c-8596-e5f2ee3fbba4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4" Oct 30 12:22:45.936333 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:45.935723 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxnrv\" (UniqueName: \"kubernetes.io/projected/a14ade7a-fdde-453c-8596-e5f2ee3fbba4-kube-api-access-jxnrv\") pod \"payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4\" (UID: \"a14ade7a-fdde-453c-8596-e5f2ee3fbba4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4" Oct 30 12:22:46.068100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:46.054986 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a14ade7a-fdde-453c-8596-e5f2ee3fbba4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4\" (UID: \"a14ade7a-fdde-453c-8596-e5f2ee3fbba4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4" Oct 30 12:22:46.068496 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:46.055981 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a14ade7a-fdde-453c-8596-e5f2ee3fbba4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4\" (UID: \"a14ade7a-fdde-453c-8596-e5f2ee3fbba4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4" Oct 30 12:22:46.084675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:46.084638 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxnrv\" (UniqueName: \"kubernetes.io/projected/a14ade7a-fdde-453c-8596-e5f2ee3fbba4-kube-api-access-jxnrv\") pod \"payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4\" (UID: \"a14ade7a-fdde-453c-8596-e5f2ee3fbba4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4" Oct 30 12:22:46.126264 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:46.126233 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxnrv\" (UniqueName: \"kubernetes.io/projected/a14ade7a-fdde-453c-8596-e5f2ee3fbba4-kube-api-access-jxnrv\") pod \"payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4\" (UID: \"a14ade7a-fdde-453c-8596-e5f2ee3fbba4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4" Oct 30 12:22:46.127655 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:46.127625 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg"] Oct 30 12:22:46.134124 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:46.132933 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg" Oct 30 12:22:46.136294 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:46.136259 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg"] Oct 30 12:22:46.288233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:46.288133 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d0a786-32e0-45f8-82bd-9874b9e26e60-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg\" (UID: \"d3d0a786-32e0-45f8-82bd-9874b9e26e60\") " pod="openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg" Oct 30 12:22:46.288406 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:46.288289 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spblr\" (UniqueName: \"kubernetes.io/projected/d3d0a786-32e0-45f8-82bd-9874b9e26e60-kube-api-access-spblr\") pod \"payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg\" (UID: \"d3d0a786-32e0-45f8-82bd-9874b9e26e60\") " pod="openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg" Oct 30 12:22:46.347881 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:46.347846 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4" Oct 30 12:22:46.403249 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:46.403115 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d0a786-32e0-45f8-82bd-9874b9e26e60-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg\" (UID: \"d3d0a786-32e0-45f8-82bd-9874b9e26e60\") " pod="openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg" Oct 30 12:22:46.405312 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:46.403297 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spblr\" (UniqueName: \"kubernetes.io/projected/d3d0a786-32e0-45f8-82bd-9874b9e26e60-kube-api-access-spblr\") pod \"payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg\" (UID: \"d3d0a786-32e0-45f8-82bd-9874b9e26e60\") " pod="openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg" Oct 30 12:22:46.405312 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:46.404805 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d0a786-32e0-45f8-82bd-9874b9e26e60-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg\" (UID: \"d3d0a786-32e0-45f8-82bd-9874b9e26e60\") " pod="openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg" Oct 30 12:22:46.442343 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:46.441969 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spblr\" (UniqueName: \"kubernetes.io/projected/d3d0a786-32e0-45f8-82bd-9874b9e26e60-kube-api-access-spblr\") pod \"payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg\" (UID: \"d3d0a786-32e0-45f8-82bd-9874b9e26e60\") " pod="openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg" Oct 30 12:22:46.457227 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:46.457193 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg" Oct 30 12:22:46.805963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:46.805911 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4"] Oct 30 12:22:46.919424 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:22:46.900710 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14ade7a_fdde_453c_8596_e5f2ee3fbba4.slice/crio-41faaad90e529df198f0f0dc191268820b1f99c63d8dbb2fc210688e28e86203 WatchSource:0}: Error finding container 41faaad90e529df198f0f0dc191268820b1f99c63d8dbb2fc210688e28e86203: Status 404 returned error can't find the container with id 41faaad90e529df198f0f0dc191268820b1f99c63d8dbb2fc210688e28e86203 Oct 30 12:22:47.233853 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.233805 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4" event={"ID":"a14ade7a-fdde-453c-8596-e5f2ee3fbba4","Type":"ContainerStarted","Data":"41faaad90e529df198f0f0dc191268820b1f99c63d8dbb2fc210688e28e86203"} Oct 30 12:22:47.234143 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.233878 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg" event={"ID":"d3d0a786-32e0-45f8-82bd-9874b9e26e60","Type":"ContainerStarted","Data":"60184bc575519a583182ebd22eb33c419acec954b2b00ad53fa7bd6bcf19a55e"} Oct 30 12:22:47.234143 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.233908 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg"] Oct 30 12:22:47.236373 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.236343 4468 generic.go:356] "Generic (PLEG): container finished" podID="8278ddb9-839e-4d62-b0b0-ba9cf036ae10" containerID="987e3f9fb2335a367e6605b2132ae3e5c58c93117883c0b6391a962ff1e5719e" exitCode=0 Oct 30 12:22:47.236515 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.236421 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" event={"ID":"8278ddb9-839e-4d62-b0b0-ba9cf036ae10","Type":"ContainerDied","Data":"987e3f9fb2335a367e6605b2132ae3e5c58c93117883c0b6391a962ff1e5719e"} Oct 30 12:22:47.236515 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.236497 4468 scope.go:122] "RemoveContainer" containerID="eb7fbf606df0f6a6e7721840290827991f20577d169918cbba8279670034acd7" Oct 30 12:22:47.244357 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.244330 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:22:47.311096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.308147 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-workload-certs\") pod \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " Oct 30 12:22:47.311096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.308238 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-envoy\") pod \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " Oct 30 12:22:47.311096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.308295 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-credential-socket\") pod \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " Oct 30 12:22:47.311096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.310166 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-workload-certs" pod "8278ddb9-839e-4d62-b0b0-ba9cf036ae10" (UID: "8278ddb9-839e-4d62-b0b0-ba9cf036ae10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:22:47.311096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.310458 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-credential-socket" pod "8278ddb9-839e-4d62-b0b0-ba9cf036ae10" (UID: "8278ddb9-839e-4d62-b0b0-ba9cf036ae10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:22:47.311096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.310659 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-token\" (UniqueName: \"kubernetes.io/projected/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-token\") pod \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " Oct 30 12:22:47.311096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.310768 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-kube-api-access-qjwqw\" (UniqueName: \"kubernetes.io/projected/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-kube-api-access-qjwqw\") pod \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " Oct 30 12:22:47.311096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.310886 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-workload-socket\") pod \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " Oct 30 12:22:47.311096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.310951 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-podinfo\") pod \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " Oct 30 12:22:47.311096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.311023 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istiod-ca-cert\") pod \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " Oct 30 12:22:47.311096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.311085 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-data\") pod \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\" (UID: \"8278ddb9-839e-4d62-b0b0-ba9cf036ae10\") " Oct 30 12:22:47.312042 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.311914 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:47.312042 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.311938 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:47.315101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.313540 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-workload-socket" pod "8278ddb9-839e-4d62-b0b0-ba9cf036ae10" (UID: "8278ddb9-839e-4d62-b0b0-ba9cf036ae10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:22:47.315101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.313992 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istiod-ca-cert" pod "8278ddb9-839e-4d62-b0b0-ba9cf036ae10" (UID: "8278ddb9-839e-4d62-b0b0-ba9cf036ae10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:22:47.316159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.315993 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-kube-api-access-qjwqw" pod "8278ddb9-839e-4d62-b0b0-ba9cf036ae10" (UID: "8278ddb9-839e-4d62-b0b0-ba9cf036ae10"). InnerVolumeSpecName "kube-api-access-qjwqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:22:47.317247 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.317205 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-envoy" pod "8278ddb9-839e-4d62-b0b0-ba9cf036ae10" (UID: "8278ddb9-839e-4d62-b0b0-ba9cf036ae10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:22:47.320169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.317774 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-data" pod "8278ddb9-839e-4d62-b0b0-ba9cf036ae10" (UID: "8278ddb9-839e-4d62-b0b0-ba9cf036ae10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:22:47.320169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.319319 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-token" pod "8278ddb9-839e-4d62-b0b0-ba9cf036ae10" (UID: "8278ddb9-839e-4d62-b0b0-ba9cf036ae10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:22:47.321603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.321580 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-podinfo" pod "8278ddb9-839e-4d62-b0b0-ba9cf036ae10" (UID: "8278ddb9-839e-4d62-b0b0-ba9cf036ae10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:22:47.415106 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.414997 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:47.415665 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.415521 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjwqw\" (UniqueName: \"kubernetes.io/projected/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-kube-api-access-qjwqw\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:47.417192 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.417164 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:47.417334 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.417194 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:47.417334 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.417209 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:47.417334 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.417225 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:47.417334 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.417241 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8278ddb9-839e-4d62-b0b0-ba9cf036ae10-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:22:47.832294 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.832205 4468 patch_prober.go:29] interesting pod/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:22:47.832294 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:47.832275 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" podUID="c9321539-0223-49a1-b3bf-765475cb0057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:22:48.245585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:48.245548 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4" event={"ID":"a14ade7a-fdde-453c-8596-e5f2ee3fbba4","Type":"ContainerStarted","Data":"bfe55ec4b27f5ddcfe9bb4e69dba2c91d55e4d81e16c2d3f2f3446f150f7dbb7"} Oct 30 12:22:48.245840 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:48.245771 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4" Oct 30 12:22:48.247631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:48.247603 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg" event={"ID":"d3d0a786-32e0-45f8-82bd-9874b9e26e60","Type":"ContainerStarted","Data":"e266567bd13390826931af0a027b3afa61d0fcece8325860cbda31e70eebe9e0"} Oct 30 12:22:48.247770 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:48.247687 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg" Oct 30 12:22:48.249813 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:48.249788 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" event={"ID":"8278ddb9-839e-4d62-b0b0-ba9cf036ae10","Type":"ContainerDied","Data":"253f841502d77edff6ca87159c51682dd0a8d21d17772c26dfbda91290e293f6"} Oct 30 12:22:48.249937 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:48.249817 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9" Oct 30 12:22:48.249937 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:48.249843 4468 scope.go:122] "RemoveContainer" containerID="987e3f9fb2335a367e6605b2132ae3e5c58c93117883c0b6391a962ff1e5719e" Oct 30 12:22:48.290509 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:48.290446 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4" podStartSLOduration=3.290428264 podStartE2EDuration="3.290428264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:48.286314938 +0000 UTC m=+1719.320453334" watchObservedRunningTime="2026-10-30 12:22:48.290428264 +0000 UTC m=+1719.324566660" Oct 30 12:22:48.328682 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:48.328615 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg" podStartSLOduration=3.328592796 podStartE2EDuration="3.328592796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:48.325671174 +0000 UTC m=+1719.359809570" watchObservedRunningTime="2026-10-30 12:22:48.328592796 +0000 UTC m=+1719.362731194" Oct 30 12:22:48.395940 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:48.395839 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9"] Oct 30 12:22:48.401917 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:48.401881 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7566f5-openshift-default-65db4b858f-6c9q9"] Oct 30 12:22:49.094372 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:49.094319 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8278ddb9-839e-4d62-b0b0-ba9cf036ae10" path="/var/lib/kubelet/pods/8278ddb9-839e-4d62-b0b0-ba9cf036ae10/volumes" Oct 30 12:22:50.267290 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:50.267254 4468 generic.go:356] "Generic (PLEG): container finished" podID="c9321539-0223-49a1-b3bf-765475cb0057" containerID="1cd10abeec7e993d9f973d6e4168a7d8ac1c6fbc2c439269018e6b6642b40c9d" exitCode=0 Oct 30 12:22:50.268190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:50.267338 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" event={"ID":"c9321539-0223-49a1-b3bf-765475cb0057","Type":"ContainerDied","Data":"1cd10abeec7e993d9f973d6e4168a7d8ac1c6fbc2c439269018e6b6642b40c9d"} Oct 30 12:22:50.268190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:50.267968 4468 scope.go:122] "RemoveContainer" containerID="1cd10abeec7e993d9f973d6e4168a7d8ac1c6fbc2c439269018e6b6642b40c9d" Oct 30 12:22:50.270003 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:50.269966 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:22:50.270158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:50.270053 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:22:50.270158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:50.270113 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:22:51.292094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:51.289892 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" event={"ID":"c9321539-0223-49a1-b3bf-765475cb0057","Type":"ContainerStarted","Data":"02bdbc9400a815d53b3c3905d3be06ea943abb8c7254843951149741140568e0"} Oct 30 12:22:52.126094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:52.125520 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:22:52.126094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:52.125663 4468 patch_prober.go:29] interesting pod/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 12:22:52.126094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:52.125710 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" podUID="c9321539-0223-49a1-b3bf-765475cb0057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 12:22:53.132265 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:53.131678 4468 patch_prober.go:29] interesting pod/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 12:22:53.132265 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:53.131743 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" podUID="c9321539-0223-49a1-b3bf-765475cb0057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 12:22:53.635185 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:22:53.635137 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:22:54.127096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:54.123787 4468 patch_prober.go:29] interesting pod/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 12:22:54.127096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:54.123857 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" podUID="c9321539-0223-49a1-b3bf-765475cb0057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 12:22:55.122992 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:55.122521 4468 patch_prober.go:29] interesting pod/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 12:22:55.122992 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:55.122594 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" podUID="c9321539-0223-49a1-b3bf-765475cb0057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 12:22:56.125093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:56.124576 4468 patch_prober.go:29] interesting pod/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 12:22:56.125093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:56.124645 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" podUID="c9321539-0223-49a1-b3bf-765475cb0057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 12:22:57.125150 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:57.125111 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:22:57.125629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:57.125600 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:22:57.126528 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:57.126507 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:22:59.261648 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:59.261616 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4" Oct 30 12:22:59.262136 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:22:59.261887 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg" Oct 30 12:23:01.329385 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:01.329344 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw"] Oct 30 12:23:01.359519 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:01.359473 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf"] Oct 30 12:23:05.809134 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:05.809092 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94"] Oct 30 12:23:06.329996 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:06.329951 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw" podUID="cb9b76ae-a152-44b0-b00c-4111be5a652d" containerName="payload-processing" containerID="cri-o://f3ca78583f3a1f0bece5543ebd7a3d5ea6f928bcabd24d320a5fa38586a7eed5" gracePeriod=25 Oct 30 12:23:06.360737 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:06.360669 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf" podUID="d961b4b0-b85f-4c77-afaa-3ae13e12ed30" containerName="payload-pre-processing" containerID="cri-o://80fcda871bed73ed4b068c38bf428826e820c8a2e83d3b8b65c558d2d1cc24e3" gracePeriod=25 Oct 30 12:23:06.702736 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:06.702695 4468 generic.go:356] "Generic (PLEG): container finished" podID="d961b4b0-b85f-4c77-afaa-3ae13e12ed30" containerID="80fcda871bed73ed4b068c38bf428826e820c8a2e83d3b8b65c558d2d1cc24e3" exitCode=0 Oct 30 12:23:06.705230 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:06.703378 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf" event={"ID":"d961b4b0-b85f-4c77-afaa-3ae13e12ed30","Type":"ContainerDied","Data":"80fcda871bed73ed4b068c38bf428826e820c8a2e83d3b8b65c558d2d1cc24e3"} Oct 30 12:23:06.708326 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:06.708288 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw" event={"ID":"cb9b76ae-a152-44b0-b00c-4111be5a652d","Type":"ContainerDied","Data":"f3ca78583f3a1f0bece5543ebd7a3d5ea6f928bcabd24d320a5fa38586a7eed5"} Oct 30 12:23:06.708537 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:06.708096 4468 generic.go:356] "Generic (PLEG): container finished" podID="cb9b76ae-a152-44b0-b00c-4111be5a652d" containerID="f3ca78583f3a1f0bece5543ebd7a3d5ea6f928bcabd24d320a5fa38586a7eed5" exitCode=0 Oct 30 12:23:06.944338 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:06.944304 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw" Oct 30 12:23:07.018414 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.018389 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf" Oct 30 12:23:07.058994 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.057465 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb9b76ae-a152-44b0-b00c-4111be5a652d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9b76ae-a152-44b0-b00c-4111be5a652d-plugins-config-volume\") pod \"cb9b76ae-a152-44b0-b00c-4111be5a652d\" (UID: \"cb9b76ae-a152-44b0-b00c-4111be5a652d\") " Oct 30 12:23:07.058994 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.057618 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb9b76ae-a152-44b0-b00c-4111be5a652d-kube-api-access-nbhdl\" (UniqueName: \"kubernetes.io/projected/cb9b76ae-a152-44b0-b00c-4111be5a652d-kube-api-access-nbhdl\") pod \"cb9b76ae-a152-44b0-b00c-4111be5a652d\" (UID: \"cb9b76ae-a152-44b0-b00c-4111be5a652d\") " Oct 30 12:23:07.058994 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.057913 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb9b76ae-a152-44b0-b00c-4111be5a652d-plugins-config-volume" pod "cb9b76ae-a152-44b0-b00c-4111be5a652d" (UID: "cb9b76ae-a152-44b0-b00c-4111be5a652d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:23:07.058994 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.058955 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9b76ae-a152-44b0-b00c-4111be5a652d-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:23:07.060384 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.060351 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9b76ae-a152-44b0-b00c-4111be5a652d-kube-api-access-nbhdl" pod "cb9b76ae-a152-44b0-b00c-4111be5a652d" (UID: "cb9b76ae-a152-44b0-b00c-4111be5a652d"). InnerVolumeSpecName "kube-api-access-nbhdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:23:07.160350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.160311 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d961b4b0-b85f-4c77-afaa-3ae13e12ed30-kube-api-access-vhtzv\" (UniqueName: \"kubernetes.io/projected/d961b4b0-b85f-4c77-afaa-3ae13e12ed30-kube-api-access-vhtzv\") pod \"d961b4b0-b85f-4c77-afaa-3ae13e12ed30\" (UID: \"d961b4b0-b85f-4c77-afaa-3ae13e12ed30\") " Oct 30 12:23:07.160558 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.160375 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d961b4b0-b85f-4c77-afaa-3ae13e12ed30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d961b4b0-b85f-4c77-afaa-3ae13e12ed30-plugins-config-volume\") pod \"d961b4b0-b85f-4c77-afaa-3ae13e12ed30\" (UID: \"d961b4b0-b85f-4c77-afaa-3ae13e12ed30\") " Oct 30 12:23:07.160802 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.160779 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbhdl\" (UniqueName: \"kubernetes.io/projected/cb9b76ae-a152-44b0-b00c-4111be5a652d-kube-api-access-nbhdl\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:23:07.160925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.160809 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d961b4b0-b85f-4c77-afaa-3ae13e12ed30-plugins-config-volume" pod "d961b4b0-b85f-4c77-afaa-3ae13e12ed30" (UID: "d961b4b0-b85f-4c77-afaa-3ae13e12ed30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:23:07.165217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.162686 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d961b4b0-b85f-4c77-afaa-3ae13e12ed30-kube-api-access-vhtzv" pod "d961b4b0-b85f-4c77-afaa-3ae13e12ed30" (UID: "d961b4b0-b85f-4c77-afaa-3ae13e12ed30"). InnerVolumeSpecName "kube-api-access-vhtzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:23:07.265434 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.262167 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhtzv\" (UniqueName: \"kubernetes.io/projected/d961b4b0-b85f-4c77-afaa-3ae13e12ed30-kube-api-access-vhtzv\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:23:07.265434 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.262204 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d961b4b0-b85f-4c77-afaa-3ae13e12ed30-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:23:07.717029 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.716996 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf" Oct 30 12:23:07.717252 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.717023 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf" event={"ID":"d961b4b0-b85f-4c77-afaa-3ae13e12ed30","Type":"ContainerDied","Data":"80bcf20326ce329f50269338ba7d5d7fac44758aee327e9ea6cdda4dd34beb49"} Oct 30 12:23:07.717252 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.717140 4468 scope.go:122] "RemoveContainer" containerID="80fcda871bed73ed4b068c38bf428826e820c8a2e83d3b8b65c558d2d1cc24e3" Oct 30 12:23:07.718693 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.718671 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw" Oct 30 12:23:07.718775 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.718695 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw" event={"ID":"cb9b76ae-a152-44b0-b00c-4111be5a652d","Type":"ContainerDied","Data":"cd7cbc6de79828d9ec8fd48a5fbe983c7d42a823a986b933cc1c7781b215fce3"} Oct 30 12:23:07.729424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.729388 4468 scope.go:122] "RemoveContainer" containerID="f3ca78583f3a1f0bece5543ebd7a3d5ea6f928bcabd24d320a5fa38586a7eed5" Oct 30 12:23:07.767870 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.767835 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf"] Oct 30 12:23:07.779281 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.779246 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-060f41f2-7d56d9b55f-52rtf"] Oct 30 12:23:07.803308 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.803267 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw"] Oct 30 12:23:07.807553 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:07.807525 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-j6gbw"] Oct 30 12:23:09.104554 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:09.104517 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb9b76ae-a152-44b0-b00c-4111be5a652d" path="/var/lib/kubelet/pods/cb9b76ae-a152-44b0-b00c-4111be5a652d/volumes" Oct 30 12:23:09.105061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:09.105041 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d961b4b0-b85f-4c77-afaa-3ae13e12ed30" path="/var/lib/kubelet/pods/d961b4b0-b85f-4c77-afaa-3ae13e12ed30/volumes" Oct 30 12:23:10.809649 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:10.809601 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94" podUID="a4ed41c3-aa61-4f83-b933-5f22e327f054" containerName="payload-pre-processing" containerID="cri-o://b1255723de1034a617790e37d103692adbb1219e1f8d177351556be6f70c3837" gracePeriod=25 Oct 30 12:23:11.119008 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:11.118974 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94" Oct 30 12:23:11.229673 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:11.229626 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4ed41c3-aa61-4f83-b933-5f22e327f054-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4ed41c3-aa61-4f83-b933-5f22e327f054-plugins-config-volume\") pod \"a4ed41c3-aa61-4f83-b933-5f22e327f054\" (UID: \"a4ed41c3-aa61-4f83-b933-5f22e327f054\") " Oct 30 12:23:11.229859 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:11.229727 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ed41c3-aa61-4f83-b933-5f22e327f054-kube-api-access-pphsq\" (UniqueName: \"kubernetes.io/projected/a4ed41c3-aa61-4f83-b933-5f22e327f054-kube-api-access-pphsq\") pod \"a4ed41c3-aa61-4f83-b933-5f22e327f054\" (UID: \"a4ed41c3-aa61-4f83-b933-5f22e327f054\") " Oct 30 12:23:11.230145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:11.230116 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ed41c3-aa61-4f83-b933-5f22e327f054-plugins-config-volume" pod "a4ed41c3-aa61-4f83-b933-5f22e327f054" (UID: "a4ed41c3-aa61-4f83-b933-5f22e327f054"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:23:11.230602 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:11.230577 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4ed41c3-aa61-4f83-b933-5f22e327f054-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:23:11.231981 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:11.231954 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ed41c3-aa61-4f83-b933-5f22e327f054-kube-api-access-pphsq" pod "a4ed41c3-aa61-4f83-b933-5f22e327f054" (UID: "a4ed41c3-aa61-4f83-b933-5f22e327f054"). InnerVolumeSpecName "kube-api-access-pphsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:23:11.332832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:11.332784 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pphsq\" (UniqueName: \"kubernetes.io/projected/a4ed41c3-aa61-4f83-b933-5f22e327f054-kube-api-access-pphsq\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:23:11.760121 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:11.757272 4468 generic.go:356] "Generic (PLEG): container finished" podID="a4ed41c3-aa61-4f83-b933-5f22e327f054" containerID="b1255723de1034a617790e37d103692adbb1219e1f8d177351556be6f70c3837" exitCode=0 Oct 30 12:23:11.760121 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:11.757331 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94" Oct 30 12:23:11.760121 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:11.757329 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94" event={"ID":"a4ed41c3-aa61-4f83-b933-5f22e327f054","Type":"ContainerDied","Data":"b1255723de1034a617790e37d103692adbb1219e1f8d177351556be6f70c3837"} Oct 30 12:23:11.760121 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:11.757432 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94" event={"ID":"a4ed41c3-aa61-4f83-b933-5f22e327f054","Type":"ContainerDied","Data":"98186d7729e68e5a6e6a567c7df84bc9787c458e4ed2547738b793ae093dbf8e"} Oct 30 12:23:11.760121 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:11.757468 4468 scope.go:122] "RemoveContainer" containerID="b1255723de1034a617790e37d103692adbb1219e1f8d177351556be6f70c3837" Oct 30 12:23:11.768719 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:11.768697 4468 scope.go:122] "RemoveContainer" containerID="b1255723de1034a617790e37d103692adbb1219e1f8d177351556be6f70c3837" Oct 30 12:23:11.769085 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:23:11.769045 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1255723de1034a617790e37d103692adbb1219e1f8d177351556be6f70c3837\": container with ID starting with b1255723de1034a617790e37d103692adbb1219e1f8d177351556be6f70c3837 not found: ID does not exist" containerID="b1255723de1034a617790e37d103692adbb1219e1f8d177351556be6f70c3837" Oct 30 12:23:11.769180 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:11.769132 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1255723de1034a617790e37d103692adbb1219e1f8d177351556be6f70c3837"} err="failed to get container status \"b1255723de1034a617790e37d103692adbb1219e1f8d177351556be6f70c3837\": rpc error: code = NotFound desc = could not find container \"b1255723de1034a617790e37d103692adbb1219e1f8d177351556be6f70c3837\": container with ID starting with b1255723de1034a617790e37d103692adbb1219e1f8d177351556be6f70c3837 not found: ID does not exist" Oct 30 12:23:11.809651 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:11.809609 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94"] Oct 30 12:23:11.821426 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:11.821390 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ebc6bb-5445fd6f99-zwd94"] Oct 30 12:23:13.093660 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:13.093623 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4ed41c3-aa61-4f83-b933-5f22e327f054" path="/var/lib/kubelet/pods/a4ed41c3-aa61-4f83-b933-5f22e327f054/volumes" Oct 30 12:23:23.605260 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:23:23.605169 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:23:31.928901 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:31.928860 4468 generic.go:356] "Generic (PLEG): container finished" podID="731fb32a-31aa-4d57-8edd-4f890b9e9288" containerID="022a351b09d8e75d8de1d71134f12cd630c9970870fc1eb5670a59e114c40232" exitCode=1 Oct 30 12:23:31.929337 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:31.928916 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" event={"ID":"731fb32a-31aa-4d57-8edd-4f890b9e9288","Type":"ContainerDied","Data":"022a351b09d8e75d8de1d71134f12cd630c9970870fc1eb5670a59e114c40232"} Oct 30 12:23:31.929385 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:31.929352 4468 scope.go:122] "RemoveContainer" containerID="022a351b09d8e75d8de1d71134f12cd630c9970870fc1eb5670a59e114c40232" Oct 30 12:23:32.352963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:32.352929 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 12:23:32.938812 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:32.938773 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" event={"ID":"731fb32a-31aa-4d57-8edd-4f890b9e9288","Type":"ContainerStarted","Data":"26047206af3ea28f4d19b69d333b3633fef7d2bf34b79f02085a3edcee891c6f"} Oct 30 12:23:32.939387 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:32.939135 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" Oct 30 12:23:43.949021 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:43.948990 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bb8f9449b-nnd2b" Oct 30 12:23:53.601152 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:23:53.601108 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:23:56.139804 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:56.139766 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt"] Oct 30 12:23:56.140421 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:56.139997 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" podUID="9f251628-d235-4066-9bb0-26067f9356db" containerName="maas-api" containerID="cri-o://d83283cb6c1abb2135c8d36ea51346a50f711c3108b8631a812bcecb70e415c6" gracePeriod=30 Oct 30 12:23:56.713333 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:56.708134 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" Oct 30 12:23:56.732024 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:56.730644 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f251628-d235-4066-9bb0-26067f9356db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f251628-d235-4066-9bb0-26067f9356db-openshift-service-ca\") pod \"9f251628-d235-4066-9bb0-26067f9356db\" (UID: \"9f251628-d235-4066-9bb0-26067f9356db\") " Oct 30 12:23:56.732024 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:56.730735 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f251628-d235-4066-9bb0-26067f9356db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f251628-d235-4066-9bb0-26067f9356db-maas-api-tls\") pod \"9f251628-d235-4066-9bb0-26067f9356db\" (UID: \"9f251628-d235-4066-9bb0-26067f9356db\") " Oct 30 12:23:56.732024 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:56.730785 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f251628-d235-4066-9bb0-26067f9356db-kube-api-access-brdbk\" (UniqueName: \"kubernetes.io/projected/9f251628-d235-4066-9bb0-26067f9356db-kube-api-access-brdbk\") pod \"9f251628-d235-4066-9bb0-26067f9356db\" (UID: \"9f251628-d235-4066-9bb0-26067f9356db\") " Oct 30 12:23:56.732024 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:56.731038 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f251628-d235-4066-9bb0-26067f9356db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f251628-d235-4066-9bb0-26067f9356db-metrics-certs\") pod \"9f251628-d235-4066-9bb0-26067f9356db\" (UID: \"9f251628-d235-4066-9bb0-26067f9356db\") " Oct 30 12:23:56.741335 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:56.741296 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f251628-d235-4066-9bb0-26067f9356db-openshift-service-ca" pod "9f251628-d235-4066-9bb0-26067f9356db" (UID: "9f251628-d235-4066-9bb0-26067f9356db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:23:56.784619 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:56.784549 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f251628-d235-4066-9bb0-26067f9356db-metrics-certs" pod "9f251628-d235-4066-9bb0-26067f9356db" (UID: "9f251628-d235-4066-9bb0-26067f9356db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:23:56.824383 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:56.824258 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f251628-d235-4066-9bb0-26067f9356db-kube-api-access-brdbk" pod "9f251628-d235-4066-9bb0-26067f9356db" (UID: "9f251628-d235-4066-9bb0-26067f9356db"). InnerVolumeSpecName "kube-api-access-brdbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:23:56.839058 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:56.833245 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f251628-d235-4066-9bb0-26067f9356db-maas-api-tls" pod "9f251628-d235-4066-9bb0-26067f9356db" (UID: "9f251628-d235-4066-9bb0-26067f9356db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:23:56.839058 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:56.837628 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f251628-d235-4066-9bb0-26067f9356db-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:23:56.839058 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:56.837657 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f251628-d235-4066-9bb0-26067f9356db-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:23:56.839058 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:56.837671 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f251628-d235-4066-9bb0-26067f9356db-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:23:56.839058 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:56.837683 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brdbk\" (UniqueName: \"kubernetes.io/projected/9f251628-d235-4066-9bb0-26067f9356db-kube-api-access-brdbk\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:23:57.177791 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:57.177536 4468 generic.go:356] "Generic (PLEG): container finished" podID="9f251628-d235-4066-9bb0-26067f9356db" containerID="d83283cb6c1abb2135c8d36ea51346a50f711c3108b8631a812bcecb70e415c6" exitCode=0 Oct 30 12:23:57.177791 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:57.177581 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" Oct 30 12:23:57.177791 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:57.177579 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" event={"ID":"9f251628-d235-4066-9bb0-26067f9356db","Type":"ContainerDied","Data":"d83283cb6c1abb2135c8d36ea51346a50f711c3108b8631a812bcecb70e415c6"} Oct 30 12:23:57.177791 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:57.177694 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt" event={"ID":"9f251628-d235-4066-9bb0-26067f9356db","Type":"ContainerDied","Data":"6e8817134bae3b54070985196ec20e1cf5296ce0b7ea469d11b8aa9c134ba14e"} Oct 30 12:23:57.177791 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:57.177725 4468 scope.go:122] "RemoveContainer" containerID="d83283cb6c1abb2135c8d36ea51346a50f711c3108b8631a812bcecb70e415c6" Oct 30 12:23:57.191579 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:57.191558 4468 scope.go:122] "RemoveContainer" containerID="d83283cb6c1abb2135c8d36ea51346a50f711c3108b8631a812bcecb70e415c6" Oct 30 12:23:57.191869 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:23:57.191849 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d83283cb6c1abb2135c8d36ea51346a50f711c3108b8631a812bcecb70e415c6\": container with ID starting with d83283cb6c1abb2135c8d36ea51346a50f711c3108b8631a812bcecb70e415c6 not found: ID does not exist" containerID="d83283cb6c1abb2135c8d36ea51346a50f711c3108b8631a812bcecb70e415c6" Oct 30 12:23:57.191972 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:57.191875 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d83283cb6c1abb2135c8d36ea51346a50f711c3108b8631a812bcecb70e415c6"} err="failed to get container status \"d83283cb6c1abb2135c8d36ea51346a50f711c3108b8631a812bcecb70e415c6\": rpc error: code = NotFound desc = could not find container \"d83283cb6c1abb2135c8d36ea51346a50f711c3108b8631a812bcecb70e415c6\": container with ID starting with d83283cb6c1abb2135c8d36ea51346a50f711c3108b8631a812bcecb70e415c6 not found: ID does not exist" Oct 30 12:23:57.277749 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:57.277634 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt"] Oct 30 12:23:57.318845 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:57.318793 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ebc6bb-f68478f57-qncdt"] Oct 30 12:23:59.091731 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:23:59.091687 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f251628-d235-4066-9bb0-26067f9356db" path="/var/lib/kubelet/pods/9f251628-d235-4066-9bb0-26067f9356db/volumes" Oct 30 12:24:05.803699 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:05.803656 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5"] Oct 30 12:24:05.804259 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:05.803965 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" podUID="2d509e5c-3337-4787-9734-3749069257ec" containerName="istio-proxy" containerID="cri-o://8684e3275ec7fa3d337330b8aaa1eae89915aed3e6a6525d32b29ddeb80b9971" gracePeriod=30 Oct 30 12:24:11.097693 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.097628 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:24:11.156751 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.156718 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d509e5c-3337-4787-9734-3749069257ec-kube-api-access-tssj7\" (UniqueName: \"kubernetes.io/projected/2d509e5c-3337-4787-9734-3749069257ec-kube-api-access-tssj7\") pod \"2d509e5c-3337-4787-9734-3749069257ec\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " Oct 30 12:24:11.156983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.156777 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-istio-envoy\") pod \"2d509e5c-3337-4787-9734-3749069257ec\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " Oct 30 12:24:11.156983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.156797 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d509e5c-3337-4787-9734-3749069257ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d509e5c-3337-4787-9734-3749069257ec-istiod-ca-cert\") pod \"2d509e5c-3337-4787-9734-3749069257ec\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " Oct 30 12:24:11.156983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.156827 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-workload-socket\") pod \"2d509e5c-3337-4787-9734-3749069257ec\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " Oct 30 12:24:11.156983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.156931 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d509e5c-3337-4787-9734-3749069257ec-istio-token\" (UniqueName: \"kubernetes.io/projected/2d509e5c-3337-4787-9734-3749069257ec-istio-token\") pod \"2d509e5c-3337-4787-9734-3749069257ec\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " Oct 30 12:24:11.157150 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.156996 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-credential-socket\") pod \"2d509e5c-3337-4787-9734-3749069257ec\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " Oct 30 12:24:11.157188 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.157147 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-istio-data\") pod \"2d509e5c-3337-4787-9734-3749069257ec\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " Oct 30 12:24:11.157188 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.157177 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-workload-socket" pod "2d509e5c-3337-4787-9734-3749069257ec" (UID: "2d509e5c-3337-4787-9734-3749069257ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:11.157287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.157241 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d509e5c-3337-4787-9734-3749069257ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d509e5c-3337-4787-9734-3749069257ec-istio-podinfo\") pod \"2d509e5c-3337-4787-9734-3749069257ec\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " Oct 30 12:24:11.157347 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.157325 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-workload-certs\") pod \"2d509e5c-3337-4787-9734-3749069257ec\" (UID: \"2d509e5c-3337-4787-9734-3749069257ec\") " Oct 30 12:24:11.157411 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.157370 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-credential-socket" pod "2d509e5c-3337-4787-9734-3749069257ec" (UID: "2d509e5c-3337-4787-9734-3749069257ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:11.157470 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.157294 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d509e5c-3337-4787-9734-3749069257ec-istiod-ca-cert" pod "2d509e5c-3337-4787-9734-3749069257ec" (UID: "2d509e5c-3337-4787-9734-3749069257ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:24:11.157628 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.157605 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-istio-data" pod "2d509e5c-3337-4787-9734-3749069257ec" (UID: "2d509e5c-3337-4787-9734-3749069257ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:11.157872 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.157851 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-workload-certs" pod "2d509e5c-3337-4787-9734-3749069257ec" (UID: "2d509e5c-3337-4787-9734-3749069257ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:11.158048 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.158028 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d509e5c-3337-4787-9734-3749069257ec-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.158048 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.158047 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.158294 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.158060 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.158294 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.158109 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.158294 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.158124 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.159562 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.159536 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-istio-envoy" pod "2d509e5c-3337-4787-9734-3749069257ec" (UID: "2d509e5c-3337-4787-9734-3749069257ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:11.159644 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.159622 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d509e5c-3337-4787-9734-3749069257ec-kube-api-access-tssj7" pod "2d509e5c-3337-4787-9734-3749069257ec" (UID: "2d509e5c-3337-4787-9734-3749069257ec"). InnerVolumeSpecName "kube-api-access-tssj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:24:11.159851 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.159830 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d509e5c-3337-4787-9734-3749069257ec-istio-token" pod "2d509e5c-3337-4787-9734-3749069257ec" (UID: "2d509e5c-3337-4787-9734-3749069257ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:24:11.159898 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.159834 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d509e5c-3337-4787-9734-3749069257ec-istio-podinfo" pod "2d509e5c-3337-4787-9734-3749069257ec" (UID: "2d509e5c-3337-4787-9734-3749069257ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:24:11.259496 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.259447 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d509e5c-3337-4787-9734-3749069257ec-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.259496 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.259489 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tssj7\" (UniqueName: \"kubernetes.io/projected/2d509e5c-3337-4787-9734-3749069257ec-kube-api-access-tssj7\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.259496 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.259503 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d509e5c-3337-4787-9734-3749069257ec-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.259745 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.259516 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d509e5c-3337-4787-9734-3749069257ec-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.469743 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.469708 4468 generic.go:356] "Generic (PLEG): container finished" podID="2d509e5c-3337-4787-9734-3749069257ec" containerID="8684e3275ec7fa3d337330b8aaa1eae89915aed3e6a6525d32b29ddeb80b9971" exitCode=0 Oct 30 12:24:11.469960 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.469755 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" Oct 30 12:24:11.469960 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.469799 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" event={"ID":"2d509e5c-3337-4787-9734-3749069257ec","Type":"ContainerDied","Data":"8684e3275ec7fa3d337330b8aaa1eae89915aed3e6a6525d32b29ddeb80b9971"} Oct 30 12:24:11.469960 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.469863 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5" event={"ID":"2d509e5c-3337-4787-9734-3749069257ec","Type":"ContainerDied","Data":"5848fe1518f4714d7d48d0cf4431f79c6dd876bb04401d881e566c32774586a1"} Oct 30 12:24:11.469960 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.469896 4468 scope.go:122] "RemoveContainer" containerID="8684e3275ec7fa3d337330b8aaa1eae89915aed3e6a6525d32b29ddeb80b9971" Oct 30 12:24:11.481529 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.481508 4468 scope.go:122] "RemoveContainer" containerID="8684e3275ec7fa3d337330b8aaa1eae89915aed3e6a6525d32b29ddeb80b9971" Oct 30 12:24:11.481831 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:24:11.481810 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8684e3275ec7fa3d337330b8aaa1eae89915aed3e6a6525d32b29ddeb80b9971\": container with ID starting with 8684e3275ec7fa3d337330b8aaa1eae89915aed3e6a6525d32b29ddeb80b9971 not found: ID does not exist" containerID="8684e3275ec7fa3d337330b8aaa1eae89915aed3e6a6525d32b29ddeb80b9971" Oct 30 12:24:11.481890 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:11.481845 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8684e3275ec7fa3d337330b8aaa1eae89915aed3e6a6525d32b29ddeb80b9971"} err="failed to get container status \"8684e3275ec7fa3d337330b8aaa1eae89915aed3e6a6525d32b29ddeb80b9971\": rpc error: code = NotFound desc = could not find container \"8684e3275ec7fa3d337330b8aaa1eae89915aed3e6a6525d32b29ddeb80b9971\": container with ID starting with 8684e3275ec7fa3d337330b8aaa1eae89915aed3e6a6525d32b29ddeb80b9971 not found: ID does not exist" Oct 30 12:24:19.528395 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:24:19.528332 4468 controller.go:251] "Failed to update lease" err="Put \"https://ab201029ee7e2438cae25c69ded34c55-2c29b25c9fc974f3.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-131-76.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:24:23.677158 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:24:23.677112 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:24:29.529290 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:24:29.529236 4468 controller.go:251] "Failed to update lease" err="Put \"https://ab201029ee7e2438cae25c69ded34c55-2c29b25c9fc974f3.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-131-76.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:24:39.530568 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:24:39.530522 4468 controller.go:251] "Failed to update lease" err="Put \"https://ab201029ee7e2438cae25c69ded34c55-2c29b25c9fc974f3.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-131-76.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:24:49.531138 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:24:49.531048 4468 controller.go:251] "Failed to update lease" err="Put \"https://ab201029ee7e2438cae25c69ded34c55-2c29b25c9fc974f3.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-131-76.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:24:51.252915 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.252806 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.252915 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.252808 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.252915 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.252854 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-512ec5-openshift-default-dockercfg-ww4j8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.252915 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.252824 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.253511 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253476 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-47fd99\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.253637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253513 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kv86z\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.253637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253516 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-c1b1e8\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.253637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253543 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.253637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253543 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-47fd99-openshift-default-dockercfg-lmqzd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.253637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253575 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.253637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253579 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"distinct-w4-512ec5-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.253637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253601 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.253637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253615 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.253637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253622 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253639 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253640 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253694 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253736 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253737 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253746 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253736 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"default-dockercfg-2qxls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253766 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-6a7148\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253775 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253771 4468 reflector.go:578] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253827 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bs7kx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253843 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253916 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253939 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lrxbh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253949 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253958 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253972 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"downloads-dockercfg-9skpx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253915 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253989 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"cacerts\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253981 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-s8v5b\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254002 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-47fd99-dockercfg-gcppk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254010 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"premium-w4-512ec5-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254016 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"distinct-2-w0-f920f7-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254011 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254026 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254020 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254032 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-n7s7n\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.253987 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"service-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.254030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254051 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rpjw7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254042 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254057 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qvjrm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254109 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254140 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254174 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254205 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c35d6b-openshift-default-dockercfg-wdrvr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254207 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6hs8x\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254229 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jjcjh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254240 4468 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254264 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254275 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254267 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254291 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-f920f7-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254295 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254308 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254315 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-qxvtf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254320 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254369 4468 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254378 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254395 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254425 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254433 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"default-dockercfg-px7j9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254456 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f920f7\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254475 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-mlkzp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254477 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xwvts\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254477 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7cm7r\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254494 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c35d6b-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254496 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254511 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254511 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254532 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254545 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254545 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254564 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254574 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254580 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-tqhfc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254595 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254611 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pxw6w\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254617 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-dockercfg-q22rn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254619 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-oauth-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254619 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254612 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"signing-key\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254640 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:24:51.254612 4468 kubelet_node_status.go:524] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-10-30T12:24:48Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-10-30T12:24:48Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-10-30T12:24:48Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-10-30T12:24:48Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-131-76.ec2.internal\": Patch \"https://ab201029ee7e2438cae25c69ded34c55-2c29b25c9fc974f3.elb.us-east-2.amazonaws.com:6443/api/v1/nodes/ip-10-0-131-76.ec2.internal/status?timeout=10s\": http2: client connection lost" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254661 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254669 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254679 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254694 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254640 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f920f7-dockercfg-k4dgv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254704 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"premium-w1-c1b1e8-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254714 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254736 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8f2898\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254750 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"default-dockercfg-2vbn8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254760 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254764 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-6a7148-openshift-default-dockercfg-49s7b\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254773 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254778 4468 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254787 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254742 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254780 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tz47t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254804 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254812 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254817 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254823 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-parameters\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254821 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f920f7-openshift-default-dockercfg-d89vp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254840 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254847 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-dx7pr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254855 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254861 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-512ec5\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254866 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254870 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-582gs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254885 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254890 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-6a7148-dockercfg-x6mql\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254899 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254891 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254918 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254928 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kv4vr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.255422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254927 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254945 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254949 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254922 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254967 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254972 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254969 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"premium-w0-f920f7-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254979 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254988 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254992 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-l54hq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.254999 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-c1b1e8-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255009 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8f2898-dockercfg-bfpc7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255019 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255045 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255048 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8pg5x\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255055 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255093 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255106 4468 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255118 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zfblq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255122 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255052 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255153 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255153 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jxkrq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255166 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-512ec5-dockercfg-sqcrs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255180 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255195 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255189 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"distinct-2-w1-c1b1e8-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255151 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255196 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255222 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255216 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255229 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255233 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255158 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sw2hc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255186 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255242 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255263 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255256 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8n4dr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255293 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tdxpc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255273 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjl8c\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255307 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255300 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255319 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lkcrv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255257 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255280 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-847hl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255318 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-c1b1e8-dockercfg-zxfvx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255347 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rtvpm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255353 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255356 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-h6q2l\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255347 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c1b1e8-openshift-default-dockercfg-k4lgw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255357 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-llkx2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255329 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"default-dockercfg-tkn4k\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255386 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-b7n4r\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255372 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255374 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255410 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255381 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"distinct-w2-47fd99-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:24:51.255342 4468 controller.go:251] "Failed to update lease" err="Put \"https://ab201029ee7e2438cae25c69ded34c55-2c29b25c9fc974f3.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-131-76.ec2.internal?timeout=10s\": http2: client connection lost" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255451 4468 controller.go:171] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255392 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255380 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:51.257954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:51.255398 4468 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 12:24:52.097093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.096439 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 12:24:52.097093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.096595 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jxkrq\"" Oct 30 12:24:52.115095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.110037 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 12:24:52.115095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.110231 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f920f7-dockercfg-k4dgv\"" Oct 30 12:24:52.115095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.110318 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 12:24:52.125200 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.119417 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.125200 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.119569 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f920f7\"" Oct 30 12:24:52.152096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.149391 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.152096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.149524 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-h6q2l\"" Oct 30 12:24:52.152096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.149607 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.152096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.149670 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.166095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.161297 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 12:24:52.166095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.161298 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rpjw7\"" Oct 30 12:24:52.166095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.161446 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 12:24:52.173096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.167325 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-47fd99\"" Oct 30 12:24:52.173096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.167459 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.173096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.167565 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-llkx2\"" Oct 30 12:24:52.173096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.167825 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6hs8x\"" Oct 30 12:24:52.195095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.181740 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 12:24:52.195095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.182046 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.212770 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.212061 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-512ec5\"" Oct 30 12:24:52.212770 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.212317 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 12:24:52.212770 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.212454 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-s8v5b\"" Oct 30 12:24:52.212770 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.212465 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.212770 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.212540 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.213465 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.213444 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8f2898-dockercfg-bfpc7\"" Oct 30 12:24:52.213576 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.213525 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tqhfc\"" Oct 30 12:24:52.222283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.222259 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lkcrv\"" Oct 30 12:24:52.231033 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.231002 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 12:24:52.234726 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.234698 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.244476 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.244439 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"premium-w1-c1b1e8-kserve-self-signed-certs\"" Oct 30 12:24:52.248166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.248138 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 12:24:52.249132 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.249106 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"default-dockercfg-2qxls\"" Oct 30 12:24:52.249485 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.249467 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rtvpm\"" Oct 30 12:24:52.257042 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.257012 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-c1b1e8\"" Oct 30 12:24:52.281279 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.281246 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 12:24:52.281469 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.281290 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.285264 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.285243 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c35d6b-serving-cert\"" Oct 30 12:24:52.290506 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.290488 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c1b1e8-openshift-default-dockercfg-k4lgw\"" Oct 30 12:24:52.297264 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.297247 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 12:24:52.300030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.300015 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.317345 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.317321 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.318188 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.318167 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 12:24:52.320833 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.320813 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.327161 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.327136 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.330330 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.330304 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"premium-w4-512ec5-kserve-self-signed-certs\"" Oct 30 12:24:52.336818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.336784 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.337980 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.337957 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jjcjh\"" Oct 30 12:24:52.341870 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.341844 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 12:24:52.345375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.345336 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"distinct-2-w1-c1b1e8-kserve-self-signed-certs\"" Oct 30 12:24:52.345375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.345365 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-n7s7n\"" Oct 30 12:24:52.352192 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.352086 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 12:24:52.366112 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.366060 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"distinct-w2-47fd99-kserve-self-signed-certs\"" Oct 30 12:24:52.369334 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.369306 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjl8c\"" Oct 30 12:24:52.371558 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.371535 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.372359 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.372342 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 12:24:52.373556 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.373543 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.410306 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.410278 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.416164 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.416143 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 12:24:52.426251 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.426059 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 12:24:52.429396 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.429374 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 12:24:52.430351 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.430335 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-c1b1e8-dockercfg-zxfvx\"" Oct 30 12:24:52.436675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.436653 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 12:24:52.438700 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.438680 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"default-dockercfg-tkn4k\"" Oct 30 12:24:52.438862 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.438840 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-mlkzp\"" Oct 30 12:24:52.440928 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.440910 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 12:24:52.445379 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.445354 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-b7n4r\"" Oct 30 12:24:52.465196 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.449381 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 12:24:52.465196 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.449466 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.465196 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.452903 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 12:24:52.465196 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.453242 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.465196 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.461016 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-dx7pr\"" Oct 30 12:24:52.465196 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.461295 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8n4dr\"" Oct 30 12:24:52.492211 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.482718 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"distinct-2-w0-f920f7-kserve-self-signed-certs\"" Oct 30 12:24:52.492211 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.482838 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.497318 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.497287 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.498122 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.498036 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 12:24:52.502630 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.500374 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.505441 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.500498 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-9skpx\"" Oct 30 12:24:52.505441 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.500595 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kv4vr\"" Oct 30 12:24:52.505441 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.500692 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 12:24:52.505441 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.502437 4468 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:24:52.505441 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.502582 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pxw6w\"" Oct 30 12:24:52.505441 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.502647 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c35d6b-openshift-default-dockercfg-wdrvr\"" Oct 30 12:24:52.505441 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.505057 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 12:24:52.512796 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.512772 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.517866 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.517841 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.522013 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.521989 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 12:24:52.522259 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.522243 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"premium-w0-f920f7-kserve-self-signed-certs\"" Oct 30 12:24:52.525559 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.525542 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 12:24:52.525717 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.525703 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8f2898\"" Oct 30 12:24:52.530309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.530282 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.530749 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.530730 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 12:24:52.530890 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.530765 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-847hl\"" Oct 30 12:24:52.535031 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.535014 4468 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:24:52.537169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.537149 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 12:24:52.547744 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.547709 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"default-dockercfg-px7j9\"" Oct 30 12:24:52.549701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.549485 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 12:24:52.554168 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.554137 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.556744 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.556701 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zfblq\"" Oct 30 12:24:52.556744 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.556751 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tdxpc\"" Oct 30 12:24:52.563954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.563924 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-512ec5-openshift-default-dockercfg-ww4j8\"" Oct 30 12:24:52.564355 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.564334 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.573053 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.573019 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 12:24:52.575047 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.575024 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-582gs\"" Oct 30 12:24:52.577144 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.577120 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.584161 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.584142 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 12:24:52.586063 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.586044 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-47fd99-dockercfg-gcppk\"" Oct 30 12:24:52.587027 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.587010 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-f920f7-serving-cert\"" Oct 30 12:24:52.591785 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.591770 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q22rn\"" Oct 30 12:24:52.592976 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.592964 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lrxbh\"" Oct 30 12:24:52.609264 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.609176 4468 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:24:52.609641 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.609620 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8pg5x\"" Oct 30 12:24:52.609719 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.609656 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 12:24:52.609719 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.609700 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.609820 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.609730 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 12:24:52.612522 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.612503 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 12:24:52.629044 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.622481 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-qxvtf\"" Oct 30 12:24:52.629044 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.626706 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kv86z\"" Oct 30 12:24:52.629044 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.626783 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 12:24:52.648648 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.648610 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.649536 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.649513 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 12:24:52.651656 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.651634 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f920f7-openshift-default-dockercfg-d89vp\"" Oct 30 12:24:52.654879 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.654831 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.655715 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.655690 4468 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:24:52.656977 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.656956 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.662243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.662220 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.670272 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.670245 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 12:24:52.670879 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.670857 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 12:24:52.676081 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.676049 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 12:24:52.677886 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.677868 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.679999 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.679984 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 12:24:52.686714 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.686694 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bs7kx\"" Oct 30 12:24:52.690149 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.690133 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 12:24:52.698977 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.698956 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xwvts\"" Oct 30 12:24:52.711142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.711122 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 12:24:52.712960 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.712945 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.718861 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.718843 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 12:24:52.719781 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.719766 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.719961 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.719947 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 12:24:52.723998 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.723981 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 12:24:52.727038 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.727021 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.729783 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.729763 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 12:24:52.734646 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.734625 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 12:24:52.738549 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.738531 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.738689 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.738677 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-6a7148\"" Oct 30 12:24:52.743956 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.743929 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 12:24:52.744137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.744123 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-6a7148-dockercfg-x6mql\"" Oct 30 12:24:52.745462 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.745437 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 12:24:52.746794 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.746776 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-c1b1e8-serving-cert\"" Oct 30 12:24:52.757971 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.757943 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-512ec5-dockercfg-sqcrs\"" Oct 30 12:24:52.760040 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.760015 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tz47t\"" Oct 30 12:24:52.769867 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.769838 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.772615 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.772591 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 12:24:52.776205 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.776185 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.779275 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.779253 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 12:24:52.782363 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.782318 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.784719 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.784701 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 12:24:52.788515 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.788472 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 12:24:52.788515 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.788476 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sw2hc\"" Oct 30 12:24:52.790956 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.790938 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-47fd99-openshift-default-dockercfg-lmqzd\"" Oct 30 12:24:52.792805 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.792788 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.810001 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.809982 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 12:24:52.810271 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.810256 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.813605 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.813585 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-6a7148-openshift-default-dockercfg-49s7b\"" Oct 30 12:24:52.814494 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.814476 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-l54hq\"" Oct 30 12:24:52.819639 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.819619 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 12:24:52.826913 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.826893 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"distinct-w4-512ec5-kserve-self-signed-certs\"" Oct 30 12:24:52.827026 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.826931 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 12:24:52.829866 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.829828 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qvjrm\"" Oct 30 12:24:52.830924 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.830902 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 12:24:52.847462 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.846918 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 12:24:52.848172 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.848142 4468 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 12:24:52.851753 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.851719 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7cm7r\"" Oct 30 12:24:52.857179 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.857142 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 12:24:52.857179 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.857167 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-hvbd5","odh-ai-gateway-infra/maas-api-revoke-sub-00c25542-k4dcf"] Oct 30 12:24:52.857397 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.857228 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb","openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2","openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd","openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv"] Oct 30 12:24:52.858325 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858296 4468 cpu_manager.go:414] "containerMap: removing container" podUID="8278ddb9-839e-4d62-b0b0-ba9cf036ae10" containerName="istio-proxy" Oct 30 12:24:52.858325 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858324 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8278ddb9-839e-4d62-b0b0-ba9cf036ae10" containerName="istio-proxy" Oct 30 12:24:52.858529 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858364 4468 cpu_manager.go:414] "containerMap: removing container" podUID="d961b4b0-b85f-4c77-afaa-3ae13e12ed30" containerName="payload-pre-processing" Oct 30 12:24:52.858529 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858372 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d961b4b0-b85f-4c77-afaa-3ae13e12ed30" containerName="payload-pre-processing" Oct 30 12:24:52.858529 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858390 4468 cpu_manager.go:414] "containerMap: removing container" podUID="9f251628-d235-4066-9bb0-26067f9356db" containerName="maas-api" Oct 30 12:24:52.858529 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858397 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f251628-d235-4066-9bb0-26067f9356db" containerName="maas-api" Oct 30 12:24:52.858529 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858412 4468 cpu_manager.go:414] "containerMap: removing container" podUID="cb9b76ae-a152-44b0-b00c-4111be5a652d" containerName="payload-processing" Oct 30 12:24:52.858529 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858420 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb9b76ae-a152-44b0-b00c-4111be5a652d" containerName="payload-processing" Oct 30 12:24:52.858529 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858436 4468 cpu_manager.go:414] "containerMap: removing container" podUID="2d509e5c-3337-4787-9734-3749069257ec" containerName="istio-proxy" Oct 30 12:24:52.858529 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858443 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d509e5c-3337-4787-9734-3749069257ec" containerName="istio-proxy" Oct 30 12:24:52.858529 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858461 4468 cpu_manager.go:414] "containerMap: removing container" podUID="8278ddb9-839e-4d62-b0b0-ba9cf036ae10" containerName="istio-proxy" Oct 30 12:24:52.858529 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858468 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8278ddb9-839e-4d62-b0b0-ba9cf036ae10" containerName="istio-proxy" Oct 30 12:24:52.858529 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858477 4468 cpu_manager.go:414] "containerMap: removing container" podUID="a4ed41c3-aa61-4f83-b933-5f22e327f054" containerName="payload-pre-processing" Oct 30 12:24:52.858529 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858486 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4ed41c3-aa61-4f83-b933-5f22e327f054" containerName="payload-pre-processing" Oct 30 12:24:52.859101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858662 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d961b4b0-b85f-4c77-afaa-3ae13e12ed30" containerName="payload-pre-processing" Oct 30 12:24:52.859101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858679 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f251628-d235-4066-9bb0-26067f9356db" containerName="maas-api" Oct 30 12:24:52.859101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858688 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb9b76ae-a152-44b0-b00c-4111be5a652d" containerName="payload-processing" Oct 30 12:24:52.859101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858699 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d509e5c-3337-4787-9734-3749069257ec" containerName="istio-proxy" Oct 30 12:24:52.859101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858709 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="8278ddb9-839e-4d62-b0b0-ba9cf036ae10" containerName="istio-proxy" Oct 30 12:24:52.859101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.858719 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4ed41c3-aa61-4f83-b933-5f22e327f054" containerName="payload-pre-processing" Oct 30 12:24:52.860837 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.860780 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 12:24:52.862927 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.862901 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:52.862927 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.862920 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"default-dockercfg-2vbn8\"" Oct 30 12:24:52.863544 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.863518 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="8278ddb9-839e-4d62-b0b0-ba9cf036ae10" containerName="istio-proxy" Oct 30 12:24:52.865606 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.865585 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 12:24:52.866487 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.866465 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 12:24:52.867282 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.867262 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:52.872122 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.872088 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb" Oct 30 12:24:52.878830 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.878802 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg","odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq","openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl","openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4"] Oct 30 12:24:52.878971 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.878907 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv" Oct 30 12:24:52.879472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.879440 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" podUID="cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37" containerName="maas-api" containerID="cri-o://930f14db1b7488531704a1ac4d62e71e964ff694bddf18d53017189cb394af02" gracePeriod=30 Oct 30 12:24:52.880231 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.880133 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" podUID="c9321539-0223-49a1-b3bf-765475cb0057" containerName="istio-proxy" containerID="cri-o://02bdbc9400a815d53b3c3905d3be06ea943abb8c7254843951149741140568e0" gracePeriod=30 Oct 30 12:24:52.881951 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.881920 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7e2db2-openshift-default-dockercfg-5bqpv\"" Oct 30 12:24:52.883955 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.883931 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-876549-openshift-default-dockercfg-sklbx\"" Oct 30 12:24:52.888029 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.888000 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-951ad1-dockercfg-8425s\"" Oct 30 12:24:52.888309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.888194 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-951ad1\"" Oct 30 12:24:52.968089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.946042 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx2s4\" (UniqueName: \"kubernetes.io/projected/51896ebb-81a0-4746-a5a1-22521ebdc85b-kube-api-access-fx2s4\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:52.968089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.959333 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-envoy\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:52.968089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.959385 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-data\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:52.968089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.959475 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-workload-socket\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:52.968089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.959578 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-token\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:52.968089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.959612 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-podinfo\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:52.968089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.959849 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-envoy\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:52.968089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.959899 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-credential-socket\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:52.968089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.959923 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb7c9be-be70-4b37-8657-95c9a1e283ca-istiod-ca-cert\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:52.968089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.959968 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-podinfo\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:52.968089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.959995 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lhf9\" (UniqueName: \"kubernetes.io/projected/efb7c9be-be70-4b37-8657-95c9a1e283ca-kube-api-access-8lhf9\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:52.968089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.960039 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-workload-socket\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:52.968089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.960109 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-workload-certs\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:52.968089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.960168 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-data\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:52.968089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.960214 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-credential-socket\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:52.968089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.960289 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-workload-certs\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:52.968089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.960314 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-token\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:52.968089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.960373 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51896ebb-81a0-4746-a5a1-22521ebdc85b-istiod-ca-cert\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:52.984957 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.983651 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-aef61e-dockercfg-p4rnp\"" Oct 30 12:24:53.001334 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:52.992428 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-aef61e\"" Oct 30 12:24:53.070583 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.070545 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-envoy\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:53.070852 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.070830 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-credential-socket\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:53.070964 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.070950 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb7c9be-be70-4b37-8657-95c9a1e283ca-istiod-ca-cert\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:53.071083 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.071052 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-podinfo\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:53.071215 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.071199 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lhf9\" (UniqueName: \"kubernetes.io/projected/efb7c9be-be70-4b37-8657-95c9a1e283ca-kube-api-access-8lhf9\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:53.071326 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.071311 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-workload-socket\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:53.071451 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.071435 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-workload-certs\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:53.071593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.071576 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d109c257-63ad-4bc4-a8f8-4877db8be90e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-aef61e-79dc689d48-558wv\" (UID: \"d109c257-63ad-4bc4-a8f8-4877db8be90e\") " pod="openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv" Oct 30 12:24:53.071716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.071700 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-data\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:53.071888 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.071867 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-credential-socket\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:53.076248 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.071474 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-credential-socket\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:53.076248 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.072291 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-workload-certs\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:53.076472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.072485 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-workload-socket\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:53.076472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.072540 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-credential-socket\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:53.076472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.072661 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-data\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:53.076472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.073532 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-envoy\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:53.076472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.075213 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-podinfo\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:53.076472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.076227 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-workload-certs\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:53.076472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.076349 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-token\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:53.076472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.076397 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51896ebb-81a0-4746-a5a1-22521ebdc85b-istiod-ca-cert\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:53.076472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.076422 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx2s4\" (UniqueName: \"kubernetes.io/projected/51896ebb-81a0-4746-a5a1-22521ebdc85b-kube-api-access-fx2s4\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:53.076472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.076470 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-envoy\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:53.076990 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.076497 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-data\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:53.076990 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.076500 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-workload-certs\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:53.076990 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.076542 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llf4k\" (UniqueName: \"kubernetes.io/projected/d109c257-63ad-4bc4-a8f8-4877db8be90e-kube-api-access-llf4k\") pod \"payload-processing-e2e-api-b-aef61e-79dc689d48-558wv\" (UID: \"d109c257-63ad-4bc4-a8f8-4877db8be90e\") " pod="openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv" Oct 30 12:24:53.076990 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.076607 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrrrh\" (UniqueName: \"kubernetes.io/projected/fbeb09c8-bb44-43ab-b2f6-44257398a5e2-kube-api-access-qrrrh\") pod \"payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb\" (UID: \"fbeb09c8-bb44-43ab-b2f6-44257398a5e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb" Oct 30 12:24:53.076990 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.076669 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-workload-socket\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:53.076990 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.076770 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-token\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:53.076990 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.076800 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-podinfo\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:53.076990 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.076850 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb7c9be-be70-4b37-8657-95c9a1e283ca-istiod-ca-cert\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:53.076990 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.076919 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-data\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:53.077645 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.077089 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51896ebb-81a0-4746-a5a1-22521ebdc85b-istiod-ca-cert\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:53.077645 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.077228 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-workload-socket\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:53.077645 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.077353 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbeb09c8-bb44-43ab-b2f6-44257398a5e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb\" (UID: \"fbeb09c8-bb44-43ab-b2f6-44257398a5e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb" Oct 30 12:24:53.079218 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.079194 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-podinfo\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:53.079973 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.079946 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-envoy\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:53.103456 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.102805 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lhf9\" (UniqueName: \"kubernetes.io/projected/efb7c9be-be70-4b37-8657-95c9a1e283ca-kube-api-access-8lhf9\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:53.107405 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.106544 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-token\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:53.113374 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.113301 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd0f4fd-f5b8-421f-bbea-f848517e8392" path="/var/lib/kubelet/pods/0dd0f4fd-f5b8-421f-bbea-f848517e8392/volumes" Oct 30 12:24:53.114546 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.114510 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d509e5c-3337-4787-9734-3749069257ec" path="/var/lib/kubelet/pods/2d509e5c-3337-4787-9734-3749069257ec/volumes" Oct 30 12:24:53.117208 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.117182 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx2s4\" (UniqueName: \"kubernetes.io/projected/51896ebb-81a0-4746-a5a1-22521ebdc85b-kube-api-access-fx2s4\") pod \"e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:53.117446 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.117427 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-token\") pod \"e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:53.126731 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.126693 4468 generic.go:356] "Generic (PLEG): container finished" podID="cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37" containerID="930f14db1b7488531704a1ac4d62e71e964ff694bddf18d53017189cb394af02" exitCode=0 Oct 30 12:24:53.127024 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.126917 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" event={"ID":"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37","Type":"ContainerDied","Data":"930f14db1b7488531704a1ac4d62e71e964ff694bddf18d53017189cb394af02"} Oct 30 12:24:53.181167 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.178786 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbeb09c8-bb44-43ab-b2f6-44257398a5e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb\" (UID: \"fbeb09c8-bb44-43ab-b2f6-44257398a5e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb" Oct 30 12:24:53.181167 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.179023 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d109c257-63ad-4bc4-a8f8-4877db8be90e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-aef61e-79dc689d48-558wv\" (UID: \"d109c257-63ad-4bc4-a8f8-4877db8be90e\") " pod="openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv" Oct 30 12:24:53.181167 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.179175 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llf4k\" (UniqueName: \"kubernetes.io/projected/d109c257-63ad-4bc4-a8f8-4877db8be90e-kube-api-access-llf4k\") pod \"payload-processing-e2e-api-b-aef61e-79dc689d48-558wv\" (UID: \"d109c257-63ad-4bc4-a8f8-4877db8be90e\") " pod="openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv" Oct 30 12:24:53.181167 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.179224 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrrrh\" (UniqueName: \"kubernetes.io/projected/fbeb09c8-bb44-43ab-b2f6-44257398a5e2-kube-api-access-qrrrh\") pod \"payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb\" (UID: \"fbeb09c8-bb44-43ab-b2f6-44257398a5e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb" Oct 30 12:24:53.181522 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.181120 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbeb09c8-bb44-43ab-b2f6-44257398a5e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb\" (UID: \"fbeb09c8-bb44-43ab-b2f6-44257398a5e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb" Oct 30 12:24:53.181941 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.181903 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:53.184802 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.184309 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d109c257-63ad-4bc4-a8f8-4877db8be90e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-aef61e-79dc689d48-558wv\" (UID: \"d109c257-63ad-4bc4-a8f8-4877db8be90e\") " pod="openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv" Oct 30 12:24:53.268254 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.268162 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrrrh\" (UniqueName: \"kubernetes.io/projected/fbeb09c8-bb44-43ab-b2f6-44257398a5e2-kube-api-access-qrrrh\") pod \"payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb\" (UID: \"fbeb09c8-bb44-43ab-b2f6-44257398a5e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb" Oct 30 12:24:53.269738 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.269709 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llf4k\" (UniqueName: \"kubernetes.io/projected/d109c257-63ad-4bc4-a8f8-4877db8be90e-kube-api-access-llf4k\") pod \"payload-processing-e2e-api-b-aef61e-79dc689d48-558wv\" (UID: \"d109c257-63ad-4bc4-a8f8-4877db8be90e\") " pod="openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv" Oct 30 12:24:53.322591 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.322359 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:53.351389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.351343 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" Oct 30 12:24:53.361834 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.361797 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb" Oct 30 12:24:53.374957 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.374401 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv" Oct 30 12:24:53.389404 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.389370 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-maas-api-tls\") pod \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\" (UID: \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\") " Oct 30 12:24:53.389685 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.389663 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-openshift-service-ca\") pod \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\" (UID: \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\") " Oct 30 12:24:53.389785 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.389762 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-kube-api-access-5kwdj\" (UniqueName: \"kubernetes.io/projected/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-kube-api-access-5kwdj\") pod \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\" (UID: \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\") " Oct 30 12:24:53.389847 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.389827 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-metrics-certs\") pod \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\" (UID: \"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37\") " Oct 30 12:24:53.390415 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.390322 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-openshift-service-ca" pod "cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37" (UID: "cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:24:53.390771 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.390531 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:53.392571 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.392529 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-kube-api-access-5kwdj" pod "cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37" (UID: "cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37"). InnerVolumeSpecName "kube-api-access-5kwdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:24:53.393462 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.393428 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-maas-api-tls" pod "cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37" (UID: "cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:24:53.393602 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.393575 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-metrics-certs" pod "cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37" (UID: "cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:24:53.495259 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.492040 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:53.495259 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.492086 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kwdj\" (UniqueName: \"kubernetes.io/projected/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-kube-api-access-5kwdj\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:53.495259 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.492102 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:53.618651 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:24:53.618606 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:24:53.771976 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:24:53.771926 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb7c9be_be70_4b37_8657_95c9a1e283ca.slice/crio-ad9ba23ae882456f5c28f921b35dc59f394ebcc0b2242c3a6a9b9c54f2348446 WatchSource:0}: Error finding container ad9ba23ae882456f5c28f921b35dc59f394ebcc0b2242c3a6a9b9c54f2348446: Status 404 returned error can't find the container with id ad9ba23ae882456f5c28f921b35dc59f394ebcc0b2242c3a6a9b9c54f2348446 Oct 30 12:24:53.788141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.786426 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:53.788141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.786526 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:53.788141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.786571 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:53.898044 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.897351 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2"] Oct 30 12:24:53.969503 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.962157 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:53.969503 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.962248 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:53.969503 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.962282 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:53.972834 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.972796 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd"] Oct 30 12:24:53.981496 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.981026 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv"] Oct 30 12:24:53.986558 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:53.986533 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb"] Oct 30 12:24:54.157174 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.157144 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" event={"ID":"cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37","Type":"ContainerDied","Data":"1ef90e4a5644c22f8c03a7348e0874e6418bec8ff9b2609144c0b9a1b097d5a1"} Oct 30 12:24:54.157373 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.157362 4468 scope.go:122] "RemoveContainer" containerID="930f14db1b7488531704a1ac4d62e71e964ff694bddf18d53017189cb394af02" Oct 30 12:24:54.157744 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.157473 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq" Oct 30 12:24:54.162509 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.159926 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv" event={"ID":"d109c257-63ad-4bc4-a8f8-4877db8be90e","Type":"ContainerStarted","Data":"cd5787c3174b7d67ac0d119cf28aeb856ab7582fe9c99bc03b0da37996373e48"} Oct 30 12:24:54.162916 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.162881 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv" event={"ID":"d109c257-63ad-4bc4-a8f8-4877db8be90e","Type":"ContainerStarted","Data":"11d194e35ab648a0b91484b531dc65f6c3464300fe40891b180eeaeb84b056ae"} Oct 30 12:24:54.163047 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.163032 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv" Oct 30 12:24:54.167289 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.167176 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb" event={"ID":"fbeb09c8-bb44-43ab-b2f6-44257398a5e2","Type":"ContainerStarted","Data":"5f75676689c6b9e99cfd4875696d0016624bbbcbd446d8c617afa8f888285767"} Oct 30 12:24:54.167289 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.167247 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb" event={"ID":"fbeb09c8-bb44-43ab-b2f6-44257398a5e2","Type":"ContainerStarted","Data":"ca5361cfc9d8f519025945dcb3c3b3bbcb1e1328a87145dd5f3dfc2f93b39a80"} Oct 30 12:24:54.167680 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.167590 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb" Oct 30 12:24:54.171133 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.170233 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" event={"ID":"51896ebb-81a0-4746-a5a1-22521ebdc85b","Type":"ContainerStarted","Data":"dc87487d2ae8cdf7ccad88fedc84bc2181ec547cd7897bc4a57b2c122dc93ec1"} Oct 30 12:24:54.171133 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.170295 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" event={"ID":"51896ebb-81a0-4746-a5a1-22521ebdc85b","Type":"ContainerStarted","Data":"f7880b2f9309690929f57453df1b1004b7c0acb99b7c728e43d212ee4a26c12c"} Oct 30 12:24:54.179543 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.179462 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" event={"ID":"efb7c9be-be70-4b37-8657-95c9a1e283ca","Type":"ContainerStarted","Data":"49918c97bcec47db9860abf33290cf90c23f9f1de00b1c6c8453578a4c4bbee0"} Oct 30 12:24:54.179543 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.179526 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" event={"ID":"efb7c9be-be70-4b37-8657-95c9a1e283ca","Type":"ContainerStarted","Data":"ad9ba23ae882456f5c28f921b35dc59f394ebcc0b2242c3a6a9b9c54f2348446"} Oct 30 12:24:54.184862 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.184832 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:54.228983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.228923 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" podStartSLOduration=40.228903342 podStartE2EDuration="40.228903342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:54.224748412 +0000 UTC m=+1845.258886828" watchObservedRunningTime="2026-10-30 12:24:54.228903342 +0000 UTC m=+1845.263041734" Oct 30 12:24:54.270414 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.270346 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb" podStartSLOduration=14.270329767 podStartE2EDuration="14.270329767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:54.26924375 +0000 UTC m=+1845.303382146" watchObservedRunningTime="2026-10-30 12:24:54.270329767 +0000 UTC m=+1845.304468163" Oct 30 12:24:54.314413 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.314356 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv" podStartSLOduration=6.314342897 podStartE2EDuration="6.314342897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:54.312406927 +0000 UTC m=+1845.346545333" watchObservedRunningTime="2026-10-30 12:24:54.314342897 +0000 UTC m=+1845.348481293" Oct 30 12:24:54.323465 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.323430 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:54.367485 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.367408 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" podStartSLOduration=46.367385336 podStartE2EDuration="46.367385336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:54.360229852 +0000 UTC m=+1845.394368261" watchObservedRunningTime="2026-10-30 12:24:54.367385336 +0000 UTC m=+1845.401523733" Oct 30 12:24:54.403013 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.402965 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq"] Oct 30 12:24:54.410737 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:54.410704 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c35d6b-5b96b7776c-6qxrq"] Oct 30 12:24:55.090771 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:55.090736 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37" path="/var/lib/kubelet/pods/cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37/volumes" Oct 30 12:24:55.197497 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:55.197466 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:55.209534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:55.209500 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:55.211821 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:55.211795 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:24:55.334159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:55.334123 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:55.800821 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:55.800756 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6"] Oct 30 12:24:55.801031 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:55.801002 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" podUID="29b6677b-eb87-4662-9b1b-2c0a4832d1d5" containerName="istio-proxy" containerID="cri-o://394b075ee687c46161d0f994ab4d474aff633fceb5cc2d632bc009192ba2560a" gracePeriod=30 Oct 30 12:24:55.806555 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:55.806528 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn"] Oct 30 12:24:55.806749 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:55.806726 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" podUID="32eeefc6-4a6c-439a-b825-fc318449dc66" containerName="istio-proxy" containerID="cri-o://4b472ff45a71f0849d987df82a0a77be300887d40a7996850261cb7a14a1ae84" gracePeriod=30 Oct 30 12:24:55.812455 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:55.812433 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw"] Oct 30 12:24:55.812629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:55.812608 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" podUID="f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" containerName="istio-proxy" containerID="cri-o://8eb50f13ee8ccc2a7795c3f8127c0eaeb7c94e63cb3b6adc4d184d8b1e9bdf59" gracePeriod=30 Oct 30 12:24:55.818351 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:55.818325 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z"] Oct 30 12:24:55.818532 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:55.818509 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" podUID="f2407a56-45e8-4257-b271-437daa98e3f4" containerName="istio-proxy" containerID="cri-o://b83f019a39ed73320794428933bd63fe967227d54f193e0d6afafa8683d82146" gracePeriod=30 Oct 30 12:24:56.212908 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:56.212861 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:56.213657 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:56.213638 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:24:57.188844 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:57.188808 4468 patch_prober.go:29] interesting pod/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:24:57.189458 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:57.189423 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" podUID="c9321539-0223-49a1-b3bf-765475cb0057" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:24:57.880357 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:57.880287 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4" podUID="a14ade7a-fdde-453c-8596-e5f2ee3fbba4" containerName="payload-pre-processing" containerID="cri-o://bfe55ec4b27f5ddcfe9bb4e69dba2c91d55e4d81e16c2d3f2f3446f150f7dbb7" gracePeriod=25 Oct 30 12:24:57.880575 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:57.880306 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg" podUID="d3d0a786-32e0-45f8-82bd-9874b9e26e60" containerName="payload-processing" containerID="cri-o://e266567bd13390826931af0a027b3afa61d0fcece8325860cbda31e70eebe9e0" gracePeriod=25 Oct 30 12:24:58.214786 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.214387 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:24:58.241395 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.241317 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4" Oct 30 12:24:58.265861 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.265823 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k"] Oct 30 12:24:58.266849 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.266815 4468 cpu_manager.go:414] "containerMap: removing container" podUID="cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37" containerName="maas-api" Oct 30 12:24:58.266849 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.266844 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37" containerName="maas-api" Oct 30 12:24:58.267159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.266869 4468 cpu_manager.go:414] "containerMap: removing container" podUID="c9321539-0223-49a1-b3bf-765475cb0057" containerName="istio-proxy" Oct 30 12:24:58.267159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.266879 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9321539-0223-49a1-b3bf-765475cb0057" containerName="istio-proxy" Oct 30 12:24:58.267159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.266908 4468 cpu_manager.go:414] "containerMap: removing container" podUID="a14ade7a-fdde-453c-8596-e5f2ee3fbba4" containerName="payload-pre-processing" Oct 30 12:24:58.267159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.266917 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a14ade7a-fdde-453c-8596-e5f2ee3fbba4" containerName="payload-pre-processing" Oct 30 12:24:58.267159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.267050 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9321539-0223-49a1-b3bf-765475cb0057" containerName="istio-proxy" Oct 30 12:24:58.267159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.267064 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a14ade7a-fdde-453c-8596-e5f2ee3fbba4" containerName="payload-pre-processing" Oct 30 12:24:58.267159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.267096 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9321539-0223-49a1-b3bf-765475cb0057" containerName="istio-proxy" Oct 30 12:24:58.267159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.267115 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd7f730a-2bc9-4f4a-8bb6-bedc1a59bc37" containerName="maas-api" Oct 30 12:24:58.272157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.272131 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.300696 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.300666 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k"] Oct 30 12:24:58.301137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.301098 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-8e3112-openshift-default-dockercfg-8dxk6\"" Oct 30 12:24:58.311327 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.311297 4468 generic.go:356] "Generic (PLEG): container finished" podID="a14ade7a-fdde-453c-8596-e5f2ee3fbba4" containerID="bfe55ec4b27f5ddcfe9bb4e69dba2c91d55e4d81e16c2d3f2f3446f150f7dbb7" exitCode=0 Oct 30 12:24:58.311472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.311358 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4" Oct 30 12:24:58.311472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.311423 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4" event={"ID":"a14ade7a-fdde-453c-8596-e5f2ee3fbba4","Type":"ContainerDied","Data":"bfe55ec4b27f5ddcfe9bb4e69dba2c91d55e4d81e16c2d3f2f3446f150f7dbb7"} Oct 30 12:24:58.311584 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.311476 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4" event={"ID":"a14ade7a-fdde-453c-8596-e5f2ee3fbba4","Type":"ContainerDied","Data":"41faaad90e529df198f0f0dc191268820b1f99c63d8dbb2fc210688e28e86203"} Oct 30 12:24:58.311584 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.311507 4468 scope.go:122] "RemoveContainer" containerID="bfe55ec4b27f5ddcfe9bb4e69dba2c91d55e4d81e16c2d3f2f3446f150f7dbb7" Oct 30 12:24:58.311782 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.311725 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg" Oct 30 12:24:58.312852 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.312833 4468 generic.go:356] "Generic (PLEG): container finished" podID="d3d0a786-32e0-45f8-82bd-9874b9e26e60" containerID="e266567bd13390826931af0a027b3afa61d0fcece8325860cbda31e70eebe9e0" exitCode=0 Oct 30 12:24:58.312985 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.312966 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg" event={"ID":"d3d0a786-32e0-45f8-82bd-9874b9e26e60","Type":"ContainerDied","Data":"e266567bd13390826931af0a027b3afa61d0fcece8325860cbda31e70eebe9e0"} Oct 30 12:24:58.313143 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.313008 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg" event={"ID":"d3d0a786-32e0-45f8-82bd-9874b9e26e60","Type":"ContainerDied","Data":"60184bc575519a583182ebd22eb33c419acec954b2b00ad53fa7bd6bcf19a55e"} Oct 30 12:24:58.315371 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.315219 4468 generic.go:356] "Generic (PLEG): container finished" podID="c9321539-0223-49a1-b3bf-765475cb0057" containerID="02bdbc9400a815d53b3c3905d3be06ea943abb8c7254843951149741140568e0" exitCode=0 Oct 30 12:24:58.315371 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.315267 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" event={"ID":"c9321539-0223-49a1-b3bf-765475cb0057","Type":"ContainerDied","Data":"02bdbc9400a815d53b3c3905d3be06ea943abb8c7254843951149741140568e0"} Oct 30 12:24:58.315371 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.315291 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" Oct 30 12:24:58.315371 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.315319 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl" event={"ID":"c9321539-0223-49a1-b3bf-765475cb0057","Type":"ContainerDied","Data":"8e8bb270efc77c4862629a99a04f651cf8e710ddb86ae5a9104cba73b75e343d"} Oct 30 12:24:58.326344 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.326315 4468 scope.go:122] "RemoveContainer" containerID="bfe55ec4b27f5ddcfe9bb4e69dba2c91d55e4d81e16c2d3f2f3446f150f7dbb7" Oct 30 12:24:58.326742 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:24:58.326721 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe55ec4b27f5ddcfe9bb4e69dba2c91d55e4d81e16c2d3f2f3446f150f7dbb7\": container with ID starting with bfe55ec4b27f5ddcfe9bb4e69dba2c91d55e4d81e16c2d3f2f3446f150f7dbb7 not found: ID does not exist" containerID="bfe55ec4b27f5ddcfe9bb4e69dba2c91d55e4d81e16c2d3f2f3446f150f7dbb7" Oct 30 12:24:58.326825 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.326749 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe55ec4b27f5ddcfe9bb4e69dba2c91d55e4d81e16c2d3f2f3446f150f7dbb7"} err="failed to get container status \"bfe55ec4b27f5ddcfe9bb4e69dba2c91d55e4d81e16c2d3f2f3446f150f7dbb7\": rpc error: code = NotFound desc = could not find container \"bfe55ec4b27f5ddcfe9bb4e69dba2c91d55e4d81e16c2d3f2f3446f150f7dbb7\": container with ID starting with bfe55ec4b27f5ddcfe9bb4e69dba2c91d55e4d81e16c2d3f2f3446f150f7dbb7 not found: ID does not exist" Oct 30 12:24:58.326825 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.326767 4468 scope.go:122] "RemoveContainer" containerID="e266567bd13390826931af0a027b3afa61d0fcece8325860cbda31e70eebe9e0" Oct 30 12:24:58.339459 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.339425 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-credential-socket\") pod \"c9321539-0223-49a1-b3bf-765475cb0057\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " Oct 30 12:24:58.339660 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.339482 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9321539-0223-49a1-b3bf-765475cb0057-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9321539-0223-49a1-b3bf-765475cb0057-istio-podinfo\") pod \"c9321539-0223-49a1-b3bf-765475cb0057\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " Oct 30 12:24:58.339660 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.339556 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a14ade7a-fdde-453c-8596-e5f2ee3fbba4-kube-api-access-jxnrv\" (UniqueName: \"kubernetes.io/projected/a14ade7a-fdde-453c-8596-e5f2ee3fbba4-kube-api-access-jxnrv\") pod \"a14ade7a-fdde-453c-8596-e5f2ee3fbba4\" (UID: \"a14ade7a-fdde-453c-8596-e5f2ee3fbba4\") " Oct 30 12:24:58.339660 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.339621 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-workload-certs\") pod \"c9321539-0223-49a1-b3bf-765475cb0057\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " Oct 30 12:24:58.339836 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.339701 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a14ade7a-fdde-453c-8596-e5f2ee3fbba4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a14ade7a-fdde-453c-8596-e5f2ee3fbba4-plugins-config-volume\") pod \"a14ade7a-fdde-453c-8596-e5f2ee3fbba4\" (UID: \"a14ade7a-fdde-453c-8596-e5f2ee3fbba4\") " Oct 30 12:24:58.339836 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.339796 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-workload-socket\") pod \"c9321539-0223-49a1-b3bf-765475cb0057\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " Oct 30 12:24:58.339927 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.339896 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-istio-data\") pod \"c9321539-0223-49a1-b3bf-765475cb0057\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " Oct 30 12:24:58.340121 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.339987 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9321539-0223-49a1-b3bf-765475cb0057-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9321539-0223-49a1-b3bf-765475cb0057-istiod-ca-cert\") pod \"c9321539-0223-49a1-b3bf-765475cb0057\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " Oct 30 12:24:58.340121 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.340059 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9321539-0223-49a1-b3bf-765475cb0057-istio-token\" (UniqueName: \"kubernetes.io/projected/c9321539-0223-49a1-b3bf-765475cb0057-istio-token\") pod \"c9321539-0223-49a1-b3bf-765475cb0057\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " Oct 30 12:24:58.340233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.340131 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9321539-0223-49a1-b3bf-765475cb0057-kube-api-access-46mtf\" (UniqueName: \"kubernetes.io/projected/c9321539-0223-49a1-b3bf-765475cb0057-kube-api-access-46mtf\") pod \"c9321539-0223-49a1-b3bf-765475cb0057\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " Oct 30 12:24:58.340233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.340171 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-istio-envoy\") pod \"c9321539-0223-49a1-b3bf-765475cb0057\" (UID: \"c9321539-0223-49a1-b3bf-765475cb0057\") " Oct 30 12:24:58.340464 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.340443 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-istio-data\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.340554 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.340538 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df475a56-3557-40c3-84bb-0e812127ae21-istiod-ca-cert\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.340636 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.340623 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df475a56-3557-40c3-84bb-0e812127ae21-istio-token\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.340735 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.340722 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-workload-socket\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.340801 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.340768 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-istio-envoy\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.340899 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.340874 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-workload-certs\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.340992 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.340976 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df475a56-3557-40c3-84bb-0e812127ae21-istio-podinfo\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.341925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.340716 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a14ade7a-fdde-453c-8596-e5f2ee3fbba4-plugins-config-volume" pod "a14ade7a-fdde-453c-8596-e5f2ee3fbba4" (UID: "a14ade7a-fdde-453c-8596-e5f2ee3fbba4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:24:58.341925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.340712 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-credential-socket" pod "c9321539-0223-49a1-b3bf-765475cb0057" (UID: "c9321539-0223-49a1-b3bf-765475cb0057"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:58.341925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.341182 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-credential-socket\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.341925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.341328 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69snw\" (UniqueName: \"kubernetes.io/projected/df475a56-3557-40c3-84bb-0e812127ae21-kube-api-access-69snw\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.341925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.341594 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a14ade7a-fdde-453c-8596-e5f2ee3fbba4-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:58.341925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.341636 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:58.341925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.341718 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-workload-socket" pod "c9321539-0223-49a1-b3bf-765475cb0057" (UID: "c9321539-0223-49a1-b3bf-765475cb0057"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:58.342920 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.342212 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-istio-data" pod "c9321539-0223-49a1-b3bf-765475cb0057" (UID: "c9321539-0223-49a1-b3bf-765475cb0057"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:58.342920 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.342421 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-workload-certs" pod "c9321539-0223-49a1-b3bf-765475cb0057" (UID: "c9321539-0223-49a1-b3bf-765475cb0057"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:58.342920 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.342624 4468 scope.go:122] "RemoveContainer" containerID="e266567bd13390826931af0a027b3afa61d0fcece8325860cbda31e70eebe9e0" Oct 30 12:24:58.343117 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.343086 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9321539-0223-49a1-b3bf-765475cb0057-istiod-ca-cert" pod "c9321539-0223-49a1-b3bf-765475cb0057" (UID: "c9321539-0223-49a1-b3bf-765475cb0057"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:24:58.344408 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:24:58.343318 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e266567bd13390826931af0a027b3afa61d0fcece8325860cbda31e70eebe9e0\": container with ID starting with e266567bd13390826931af0a027b3afa61d0fcece8325860cbda31e70eebe9e0 not found: ID does not exist" containerID="e266567bd13390826931af0a027b3afa61d0fcece8325860cbda31e70eebe9e0" Oct 30 12:24:58.344408 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.343359 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e266567bd13390826931af0a027b3afa61d0fcece8325860cbda31e70eebe9e0"} err="failed to get container status \"e266567bd13390826931af0a027b3afa61d0fcece8325860cbda31e70eebe9e0\": rpc error: code = NotFound desc = could not find container \"e266567bd13390826931af0a027b3afa61d0fcece8325860cbda31e70eebe9e0\": container with ID starting with e266567bd13390826931af0a027b3afa61d0fcece8325860cbda31e70eebe9e0 not found: ID does not exist" Oct 30 12:24:58.344408 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.343388 4468 scope.go:122] "RemoveContainer" containerID="02bdbc9400a815d53b3c3905d3be06ea943abb8c7254843951149741140568e0" Oct 30 12:24:58.344408 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.344219 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a14ade7a-fdde-453c-8596-e5f2ee3fbba4-kube-api-access-jxnrv" pod "a14ade7a-fdde-453c-8596-e5f2ee3fbba4" (UID: "a14ade7a-fdde-453c-8596-e5f2ee3fbba4"). InnerVolumeSpecName "kube-api-access-jxnrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:24:58.345058 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.345030 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9321539-0223-49a1-b3bf-765475cb0057-kube-api-access-46mtf" pod "c9321539-0223-49a1-b3bf-765475cb0057" (UID: "c9321539-0223-49a1-b3bf-765475cb0057"). InnerVolumeSpecName "kube-api-access-46mtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:24:58.348637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.348582 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9321539-0223-49a1-b3bf-765475cb0057-istio-podinfo" pod "c9321539-0223-49a1-b3bf-765475cb0057" (UID: "c9321539-0223-49a1-b3bf-765475cb0057"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:24:58.349113 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.349056 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9321539-0223-49a1-b3bf-765475cb0057-istio-token" pod "c9321539-0223-49a1-b3bf-765475cb0057" (UID: "c9321539-0223-49a1-b3bf-765475cb0057"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:24:58.352110 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.352042 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-istio-envoy" pod "c9321539-0223-49a1-b3bf-765475cb0057" (UID: "c9321539-0223-49a1-b3bf-765475cb0057"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:58.359782 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.359748 4468 scope.go:122] "RemoveContainer" containerID="1cd10abeec7e993d9f973d6e4168a7d8ac1c6fbc2c439269018e6b6642b40c9d" Oct 30 12:24:58.377709 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.374127 4468 scope.go:122] "RemoveContainer" containerID="02bdbc9400a815d53b3c3905d3be06ea943abb8c7254843951149741140568e0" Oct 30 12:24:58.377958 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:24:58.377924 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02bdbc9400a815d53b3c3905d3be06ea943abb8c7254843951149741140568e0\": container with ID starting with 02bdbc9400a815d53b3c3905d3be06ea943abb8c7254843951149741140568e0 not found: ID does not exist" containerID="02bdbc9400a815d53b3c3905d3be06ea943abb8c7254843951149741140568e0" Oct 30 12:24:58.378229 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.378192 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02bdbc9400a815d53b3c3905d3be06ea943abb8c7254843951149741140568e0"} err="failed to get container status \"02bdbc9400a815d53b3c3905d3be06ea943abb8c7254843951149741140568e0\": rpc error: code = NotFound desc = could not find container \"02bdbc9400a815d53b3c3905d3be06ea943abb8c7254843951149741140568e0\": container with ID starting with 02bdbc9400a815d53b3c3905d3be06ea943abb8c7254843951149741140568e0 not found: ID does not exist" Oct 30 12:24:58.378502 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.378232 4468 scope.go:122] "RemoveContainer" containerID="1cd10abeec7e993d9f973d6e4168a7d8ac1c6fbc2c439269018e6b6642b40c9d" Oct 30 12:24:58.379229 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:24:58.379200 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd10abeec7e993d9f973d6e4168a7d8ac1c6fbc2c439269018e6b6642b40c9d\": container with ID starting with 1cd10abeec7e993d9f973d6e4168a7d8ac1c6fbc2c439269018e6b6642b40c9d not found: ID does not exist" containerID="1cd10abeec7e993d9f973d6e4168a7d8ac1c6fbc2c439269018e6b6642b40c9d" Oct 30 12:24:58.379345 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.379238 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd10abeec7e993d9f973d6e4168a7d8ac1c6fbc2c439269018e6b6642b40c9d"} err="failed to get container status \"1cd10abeec7e993d9f973d6e4168a7d8ac1c6fbc2c439269018e6b6642b40c9d\": rpc error: code = NotFound desc = could not find container \"1cd10abeec7e993d9f973d6e4168a7d8ac1c6fbc2c439269018e6b6642b40c9d\": container with ID starting with 1cd10abeec7e993d9f973d6e4168a7d8ac1c6fbc2c439269018e6b6642b40c9d not found: ID does not exist" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.442812 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d0a786-32e0-45f8-82bd-9874b9e26e60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d0a786-32e0-45f8-82bd-9874b9e26e60-plugins-config-volume\") pod \"d3d0a786-32e0-45f8-82bd-9874b9e26e60\" (UID: \"d3d0a786-32e0-45f8-82bd-9874b9e26e60\") " Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.442886 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d0a786-32e0-45f8-82bd-9874b9e26e60-kube-api-access-spblr\" (UniqueName: \"kubernetes.io/projected/d3d0a786-32e0-45f8-82bd-9874b9e26e60-kube-api-access-spblr\") pod \"d3d0a786-32e0-45f8-82bd-9874b9e26e60\" (UID: \"d3d0a786-32e0-45f8-82bd-9874b9e26e60\") " Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.443414 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-credential-socket\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.443478 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69snw\" (UniqueName: \"kubernetes.io/projected/df475a56-3557-40c3-84bb-0e812127ae21-kube-api-access-69snw\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.443595 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-istio-data\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.443692 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df475a56-3557-40c3-84bb-0e812127ae21-istiod-ca-cert\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.443760 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df475a56-3557-40c3-84bb-0e812127ae21-istio-token\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.443874 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-workload-socket\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.443924 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-istio-envoy\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.444035 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-workload-certs\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.444144 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df475a56-3557-40c3-84bb-0e812127ae21-istio-podinfo\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.444278 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.444297 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.444312 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9321539-0223-49a1-b3bf-765475cb0057-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.444326 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9321539-0223-49a1-b3bf-765475cb0057-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.444341 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46mtf\" (UniqueName: \"kubernetes.io/projected/c9321539-0223-49a1-b3bf-765475cb0057-kube-api-access-46mtf\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.444355 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.444371 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9321539-0223-49a1-b3bf-765475cb0057-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.444385 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxnrv\" (UniqueName: \"kubernetes.io/projected/a14ade7a-fdde-453c-8596-e5f2ee3fbba4-kube-api-access-jxnrv\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.444401 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9321539-0223-49a1-b3bf-765475cb0057-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.459263 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d0a786-32e0-45f8-82bd-9874b9e26e60-plugins-config-volume" pod "d3d0a786-32e0-45f8-82bd-9874b9e26e60" (UID: "d3d0a786-32e0-45f8-82bd-9874b9e26e60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.481653 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-credential-socket\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.493766 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-istio-data\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.502984 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df475a56-3557-40c3-84bb-0e812127ae21-istiod-ca-cert\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.503754 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-workload-certs\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.505723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.503799 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-workload-socket\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.525471 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.523168 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df475a56-3557-40c3-84bb-0e812127ae21-istio-podinfo\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.553531 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.545226 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d0a786-32e0-45f8-82bd-9874b9e26e60-kube-api-access-spblr" pod "d3d0a786-32e0-45f8-82bd-9874b9e26e60" (UID: "d3d0a786-32e0-45f8-82bd-9874b9e26e60"). InnerVolumeSpecName "kube-api-access-spblr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:24:58.587690 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.582020 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d0a786-32e0-45f8-82bd-9874b9e26e60-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:58.587690 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.582128 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spblr\" (UniqueName: \"kubernetes.io/projected/d3d0a786-32e0-45f8-82bd-9874b9e26e60-kube-api-access-spblr\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:24:58.597479 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.595822 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69snw\" (UniqueName: \"kubernetes.io/projected/df475a56-3557-40c3-84bb-0e812127ae21-kube-api-access-69snw\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.602116 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.597729 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-istio-envoy\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.799631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.799532 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df475a56-3557-40c3-84bb-0e812127ae21-istio-token\") pod \"e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.817791 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.808996 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl"] Oct 30 12:24:58.817791 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.816006 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c35d6b-openshift-default-6865944b67-m8jvl"] Oct 30 12:24:58.851742 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.851699 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4"] Oct 30 12:24:58.858749 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.858708 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8f2898-c59b9c6f5-lx6d4"] Oct 30 12:24:58.898958 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.898923 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:24:58.906629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.905446 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx"] Oct 30 12:24:58.906629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.906437 4468 cpu_manager.go:414] "containerMap: removing container" podUID="c9321539-0223-49a1-b3bf-765475cb0057" containerName="istio-proxy" Oct 30 12:24:58.906629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.906462 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9321539-0223-49a1-b3bf-765475cb0057" containerName="istio-proxy" Oct 30 12:24:58.906629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.906521 4468 cpu_manager.go:414] "containerMap: removing container" podUID="d3d0a786-32e0-45f8-82bd-9874b9e26e60" containerName="payload-processing" Oct 30 12:24:58.906629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.906530 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d0a786-32e0-45f8-82bd-9874b9e26e60" containerName="payload-processing" Oct 30 12:24:58.906944 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.906678 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d0a786-32e0-45f8-82bd-9874b9e26e60" containerName="payload-processing" Oct 30 12:24:58.911031 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.910992 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx" Oct 30 12:24:58.929417 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.929368 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-876549\"" Oct 30 12:24:58.934354 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.933568 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx"] Oct 30 12:24:58.962092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:58.949476 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-876549-dockercfg-8gltd\"" Oct 30 12:24:59.045152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.040664 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fc01cbd-3d3d-46db-81e9-67dc614b7205-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx\" (UID: \"3fc01cbd-3d3d-46db-81e9-67dc614b7205\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx" Oct 30 12:24:59.045152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.040748 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhjcm\" (UniqueName: \"kubernetes.io/projected/3fc01cbd-3d3d-46db-81e9-67dc614b7205-kube-api-access-fhjcm\") pod \"payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx\" (UID: \"3fc01cbd-3d3d-46db-81e9-67dc614b7205\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx" Oct 30 12:24:59.222164 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.166114 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fc01cbd-3d3d-46db-81e9-67dc614b7205-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx\" (UID: \"3fc01cbd-3d3d-46db-81e9-67dc614b7205\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx" Oct 30 12:24:59.222164 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.166202 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhjcm\" (UniqueName: \"kubernetes.io/projected/3fc01cbd-3d3d-46db-81e9-67dc614b7205-kube-api-access-fhjcm\") pod \"payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx\" (UID: \"3fc01cbd-3d3d-46db-81e9-67dc614b7205\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx" Oct 30 12:24:59.222164 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.167816 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fc01cbd-3d3d-46db-81e9-67dc614b7205-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx\" (UID: \"3fc01cbd-3d3d-46db-81e9-67dc614b7205\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx" Oct 30 12:24:59.318397 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.316799 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a14ade7a-fdde-453c-8596-e5f2ee3fbba4" path="/var/lib/kubelet/pods/a14ade7a-fdde-453c-8596-e5f2ee3fbba4/volumes" Oct 30 12:24:59.328395 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.326669 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhjcm\" (UniqueName: \"kubernetes.io/projected/3fc01cbd-3d3d-46db-81e9-67dc614b7205-kube-api-access-fhjcm\") pod \"payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx\" (UID: \"3fc01cbd-3d3d-46db-81e9-67dc614b7205\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx" Oct 30 12:24:59.329120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.329083 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9321539-0223-49a1-b3bf-765475cb0057" path="/var/lib/kubelet/pods/c9321539-0223-49a1-b3bf-765475cb0057/volumes" Oct 30 12:24:59.329912 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.329885 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6"] Oct 30 12:24:59.340027 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.340003 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6" Oct 30 12:24:59.364904 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.364862 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg" Oct 30 12:24:59.367527 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.367489 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6"] Oct 30 12:24:59.369707 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.369675 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7zp6\" (UniqueName: \"kubernetes.io/projected/4bfe187d-97e5-45cb-b643-4a8946b59df1-kube-api-access-s7zp6\") pod \"payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6\" (UID: \"4bfe187d-97e5-45cb-b643-4a8946b59df1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6" Oct 30 12:24:59.370576 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.370545 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bfe187d-97e5-45cb-b643-4a8946b59df1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6\" (UID: \"4bfe187d-97e5-45cb-b643-4a8946b59df1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6" Oct 30 12:24:59.435839 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.435803 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg"] Oct 30 12:24:59.443745 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.443708 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8f2898-d5b87fd6b-2gqwg"] Oct 30 12:24:59.473052 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.473002 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7zp6\" (UniqueName: \"kubernetes.io/projected/4bfe187d-97e5-45cb-b643-4a8946b59df1-kube-api-access-s7zp6\") pod \"payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6\" (UID: \"4bfe187d-97e5-45cb-b643-4a8946b59df1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6" Oct 30 12:24:59.473242 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.473208 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bfe187d-97e5-45cb-b643-4a8946b59df1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6\" (UID: \"4bfe187d-97e5-45cb-b643-4a8946b59df1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6" Oct 30 12:24:59.473936 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.473909 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bfe187d-97e5-45cb-b643-4a8946b59df1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6\" (UID: \"4bfe187d-97e5-45cb-b643-4a8946b59df1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6" Oct 30 12:24:59.493698 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:24:59.493667 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf475a56_3557_40c3_84bb_0e812127ae21.slice/crio-e4dbfb084f93df699b431b9bc545df6e714ff33ab10400b69746ba39dbf5d431 WatchSource:0}: Error finding container e4dbfb084f93df699b431b9bc545df6e714ff33ab10400b69746ba39dbf5d431: Status 404 returned error can't find the container with id e4dbfb084f93df699b431b9bc545df6e714ff33ab10400b69746ba39dbf5d431 Oct 30 12:24:59.496334 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.496302 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:59.496427 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.496408 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:59.496495 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.496456 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:59.498983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.498958 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k"] Oct 30 12:24:59.499304 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.499290 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7zp6\" (UniqueName: \"kubernetes.io/projected/4bfe187d-97e5-45cb-b643-4a8946b59df1-kube-api-access-s7zp6\") pod \"payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6\" (UID: \"4bfe187d-97e5-45cb-b643-4a8946b59df1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6" Oct 30 12:24:59.615793 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.615750 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx" Oct 30 12:24:59.674017 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.673660 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6" Oct 30 12:24:59.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:24:59.994477 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx"] Oct 30 12:25:00.133215 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:25:00.133136 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfe187d_97e5_45cb_b643_4a8946b59df1.slice/crio-38689b86eb376b139d8a61443a087ba62a522262df4eed5f7c1c571d7d4c6283 WatchSource:0}: Error finding container 38689b86eb376b139d8a61443a087ba62a522262df4eed5f7c1c571d7d4c6283: Status 404 returned error can't find the container with id 38689b86eb376b139d8a61443a087ba62a522262df4eed5f7c1c571d7d4c6283 Oct 30 12:25:00.135806 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.135780 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6"] Oct 30 12:25:00.373377 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.373332 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6" event={"ID":"4bfe187d-97e5-45cb-b643-4a8946b59df1","Type":"ContainerStarted","Data":"dcb32b1ca11dd741a80c6666c6d8b2199746e7c2079f0e4572615ecee2067fef"} Oct 30 12:25:00.373943 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.373395 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6" event={"ID":"4bfe187d-97e5-45cb-b643-4a8946b59df1","Type":"ContainerStarted","Data":"38689b86eb376b139d8a61443a087ba62a522262df4eed5f7c1c571d7d4c6283"} Oct 30 12:25:00.373943 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.373576 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6" Oct 30 12:25:00.375236 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.375208 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx" event={"ID":"3fc01cbd-3d3d-46db-81e9-67dc614b7205","Type":"ContainerStarted","Data":"2fb9b8e7e8e2dd3bcffb357ef333f590ddd908046c47f97730f9bcfb268f09e3"} Oct 30 12:25:00.375375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.375254 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx" event={"ID":"3fc01cbd-3d3d-46db-81e9-67dc614b7205","Type":"ContainerStarted","Data":"74b70c9404ea44f14c0c71354855ad15460b34e34a31e605fb191a187c20492d"} Oct 30 12:25:00.375375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.375303 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx" Oct 30 12:25:00.376766 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.376739 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" event={"ID":"df475a56-3557-40c3-84bb-0e812127ae21","Type":"ContainerStarted","Data":"00f9b05fec387882161ab7e5a4de28ffd969d08326e5a0c690468e03a4d7a65a"} Oct 30 12:25:00.376869 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.376791 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" event={"ID":"df475a56-3557-40c3-84bb-0e812127ae21","Type":"ContainerStarted","Data":"e4dbfb084f93df699b431b9bc545df6e714ff33ab10400b69746ba39dbf5d431"} Oct 30 12:25:00.422137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.422053 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6" podStartSLOduration=2.4220355319999998 podStartE2EDuration="2.422035532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:00.418737287 +0000 UTC m=+1851.452875676" watchObservedRunningTime="2026-10-30 12:25:00.422035532 +0000 UTC m=+1851.456173931" Oct 30 12:25:00.468833 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.468769 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx" podStartSLOduration=2.468752655 podStartE2EDuration="2.468752655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:00.464913383 +0000 UTC m=+1851.499051779" watchObservedRunningTime="2026-10-30 12:25:00.468752655 +0000 UTC m=+1851.502891051" Oct 30 12:25:00.513155 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.513105 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" podStartSLOduration=2.513055798 podStartE2EDuration="2.513055798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:00.512465824 +0000 UTC m=+1851.546604221" watchObservedRunningTime="2026-10-30 12:25:00.513055798 +0000 UTC m=+1851.547194194" Oct 30 12:25:00.619885 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.619851 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz"] Oct 30 12:25:00.624372 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.624349 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz" Oct 30 12:25:00.642847 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.642810 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7e2db2\"" Oct 30 12:25:00.650572 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.650526 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz"] Oct 30 12:25:00.661858 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.661818 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7e2db2-dockercfg-fs857\"" Oct 30 12:25:00.684396 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.684306 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn"] Oct 30 12:25:00.690055 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.690023 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn" Oct 30 12:25:00.696422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.696390 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27bdw\" (UniqueName: \"kubernetes.io/projected/82ada6d9-8378-48b5-aac2-38294aa6ef5c-kube-api-access-27bdw\") pod \"payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn\" (UID: \"82ada6d9-8378-48b5-aac2-38294aa6ef5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn" Oct 30 12:25:00.696571 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.696516 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a13adda-0849-4688-a331-04b6701a8325-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz\" (UID: \"6a13adda-0849-4688-a331-04b6701a8325\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz" Oct 30 12:25:00.696571 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.696562 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ada6d9-8378-48b5-aac2-38294aa6ef5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn\" (UID: \"82ada6d9-8378-48b5-aac2-38294aa6ef5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn" Oct 30 12:25:00.696701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.696610 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9sj5\" (UniqueName: \"kubernetes.io/projected/6a13adda-0849-4688-a331-04b6701a8325-kube-api-access-z9sj5\") pod \"payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz\" (UID: \"6a13adda-0849-4688-a331-04b6701a8325\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz" Oct 30 12:25:00.711728 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.711700 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn"] Oct 30 12:25:00.806632 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.802427 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27bdw\" (UniqueName: \"kubernetes.io/projected/82ada6d9-8378-48b5-aac2-38294aa6ef5c-kube-api-access-27bdw\") pod \"payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn\" (UID: \"82ada6d9-8378-48b5-aac2-38294aa6ef5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn" Oct 30 12:25:00.806632 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.802587 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a13adda-0849-4688-a331-04b6701a8325-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz\" (UID: \"6a13adda-0849-4688-a331-04b6701a8325\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz" Oct 30 12:25:00.806632 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.802672 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ada6d9-8378-48b5-aac2-38294aa6ef5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn\" (UID: \"82ada6d9-8378-48b5-aac2-38294aa6ef5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn" Oct 30 12:25:00.806632 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.802816 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9sj5\" (UniqueName: \"kubernetes.io/projected/6a13adda-0849-4688-a331-04b6701a8325-kube-api-access-z9sj5\") pod \"payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz\" (UID: \"6a13adda-0849-4688-a331-04b6701a8325\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz" Oct 30 12:25:00.806632 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.804381 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a13adda-0849-4688-a331-04b6701a8325-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz\" (UID: \"6a13adda-0849-4688-a331-04b6701a8325\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz" Oct 30 12:25:00.835462 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.835420 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ada6d9-8378-48b5-aac2-38294aa6ef5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn\" (UID: \"82ada6d9-8378-48b5-aac2-38294aa6ef5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn" Oct 30 12:25:00.902018 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.901984 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:25:00.916429 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:00.916395 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27bdw\" (UniqueName: \"kubernetes.io/projected/82ada6d9-8378-48b5-aac2-38294aa6ef5c-kube-api-access-27bdw\") pod \"payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn\" (UID: \"82ada6d9-8378-48b5-aac2-38294aa6ef5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn" Oct 30 12:25:01.032972 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.030606 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:25:01.032972 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.031193 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn" Oct 30 12:25:01.088978 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.088939 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9sj5\" (UniqueName: \"kubernetes.io/projected/6a13adda-0849-4688-a331-04b6701a8325-kube-api-access-z9sj5\") pod \"payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz\" (UID: \"6a13adda-0849-4688-a331-04b6701a8325\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz" Oct 30 12:25:01.120642 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.119445 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d0a786-32e0-45f8-82bd-9874b9e26e60" path="/var/lib/kubelet/pods/d3d0a786-32e0-45f8-82bd-9874b9e26e60/volumes" Oct 30 12:25:01.243662 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.243622 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz" Oct 30 12:25:01.349616 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.349584 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:25:01.357788 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.357758 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:25:01.388387 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.388352 4468 generic.go:356] "Generic (PLEG): container finished" podID="29b6677b-eb87-4662-9b1b-2c0a4832d1d5" containerID="394b075ee687c46161d0f994ab4d474aff633fceb5cc2d632bc009192ba2560a" exitCode=0 Oct 30 12:25:01.388770 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.388392 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" event={"ID":"29b6677b-eb87-4662-9b1b-2c0a4832d1d5","Type":"ContainerDied","Data":"394b075ee687c46161d0f994ab4d474aff633fceb5cc2d632bc009192ba2560a"} Oct 30 12:25:01.390274 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.390247 4468 generic.go:356] "Generic (PLEG): container finished" podID="f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" containerID="8eb50f13ee8ccc2a7795c3f8127c0eaeb7c94e63cb3b6adc4d184d8b1e9bdf59" exitCode=0 Oct 30 12:25:01.390406 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.390370 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" event={"ID":"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3","Type":"ContainerDied","Data":"8eb50f13ee8ccc2a7795c3f8127c0eaeb7c94e63cb3b6adc4d184d8b1e9bdf59"} Oct 30 12:25:01.390453 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.390418 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" event={"ID":"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3","Type":"ContainerDied","Data":"e73b2d59329d564c4544bb19b1ee46568ba26eb4fef2b995bff8a3189995fd3b"} Oct 30 12:25:01.390453 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.390445 4468 scope.go:122] "RemoveContainer" containerID="8eb50f13ee8ccc2a7795c3f8127c0eaeb7c94e63cb3b6adc4d184d8b1e9bdf59" Oct 30 12:25:01.390598 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.390585 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw" Oct 30 12:25:01.393201 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.393141 4468 generic.go:356] "Generic (PLEG): container finished" podID="32eeefc6-4a6c-439a-b825-fc318449dc66" containerID="4b472ff45a71f0849d987df82a0a77be300887d40a7996850261cb7a14a1ae84" exitCode=0 Oct 30 12:25:01.393201 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.393196 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" event={"ID":"32eeefc6-4a6c-439a-b825-fc318449dc66","Type":"ContainerDied","Data":"4b472ff45a71f0849d987df82a0a77be300887d40a7996850261cb7a14a1ae84"} Oct 30 12:25:01.395349 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.395149 4468 generic.go:356] "Generic (PLEG): container finished" podID="f2407a56-45e8-4257-b271-437daa98e3f4" containerID="b83f019a39ed73320794428933bd63fe967227d54f193e0d6afafa8683d82146" exitCode=0 Oct 30 12:25:01.395349 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.395204 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" Oct 30 12:25:01.395349 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.395255 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" event={"ID":"f2407a56-45e8-4257-b271-437daa98e3f4","Type":"ContainerDied","Data":"b83f019a39ed73320794428933bd63fe967227d54f193e0d6afafa8683d82146"} Oct 30 12:25:01.395349 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.395289 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z" event={"ID":"f2407a56-45e8-4257-b271-437daa98e3f4","Type":"ContainerDied","Data":"9e17646bdf11af84e9f116c60593cfe3dfb6a7155cf92273c5eeea7251a80b7d"} Oct 30 12:25:01.402620 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.398712 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:25:01.403461 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.403439 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:25:01.444504 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.444093 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn"] Oct 30 12:25:01.444504 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.444267 4468 scope.go:122] "RemoveContainer" containerID="8eb50f13ee8ccc2a7795c3f8127c0eaeb7c94e63cb3b6adc4d184d8b1e9bdf59" Oct 30 12:25:01.444896 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:25:01.444853 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb50f13ee8ccc2a7795c3f8127c0eaeb7c94e63cb3b6adc4d184d8b1e9bdf59\": container with ID starting with 8eb50f13ee8ccc2a7795c3f8127c0eaeb7c94e63cb3b6adc4d184d8b1e9bdf59 not found: ID does not exist" containerID="8eb50f13ee8ccc2a7795c3f8127c0eaeb7c94e63cb3b6adc4d184d8b1e9bdf59" Oct 30 12:25:01.447190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.447148 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb50f13ee8ccc2a7795c3f8127c0eaeb7c94e63cb3b6adc4d184d8b1e9bdf59"} err="failed to get container status \"8eb50f13ee8ccc2a7795c3f8127c0eaeb7c94e63cb3b6adc4d184d8b1e9bdf59\": rpc error: code = NotFound desc = could not find container \"8eb50f13ee8ccc2a7795c3f8127c0eaeb7c94e63cb3b6adc4d184d8b1e9bdf59\": container with ID starting with 8eb50f13ee8ccc2a7795c3f8127c0eaeb7c94e63cb3b6adc4d184d8b1e9bdf59 not found: ID does not exist" Oct 30 12:25:01.447190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.447188 4468 scope.go:122] "RemoveContainer" containerID="b83f019a39ed73320794428933bd63fe967227d54f193e0d6afafa8683d82146" Oct 30 12:25:01.465834 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.465415 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-workload-certs\") pod \"f2407a56-45e8-4257-b271-437daa98e3f4\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " Oct 30 12:25:01.465834 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.465482 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2407a56-45e8-4257-b271-437daa98e3f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2407a56-45e8-4257-b271-437daa98e3f4-istiod-ca-cert\") pod \"f2407a56-45e8-4257-b271-437daa98e3f4\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " Oct 30 12:25:01.465834 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.465510 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2407a56-45e8-4257-b271-437daa98e3f4-kube-api-access-2vkdx\" (UniqueName: \"kubernetes.io/projected/f2407a56-45e8-4257-b271-437daa98e3f4-kube-api-access-2vkdx\") pod \"f2407a56-45e8-4257-b271-437daa98e3f4\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " Oct 30 12:25:01.465834 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.465542 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-istio-data\") pod \"f2407a56-45e8-4257-b271-437daa98e3f4\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " Oct 30 12:25:01.465834 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.465568 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-istio-envoy\") pod \"f2407a56-45e8-4257-b271-437daa98e3f4\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " Oct 30 12:25:01.465834 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.465647 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-credential-socket\") pod \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " Oct 30 12:25:01.465834 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.465694 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-workload-socket\") pod \"f2407a56-45e8-4257-b271-437daa98e3f4\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " Oct 30 12:25:01.465834 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.465724 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-envoy\") pod \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " Oct 30 12:25:01.466609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.466006 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-istio-data" pod "f2407a56-45e8-4257-b271-437daa98e3f4" (UID: "f2407a56-45e8-4257-b271-437daa98e3f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.466609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.466217 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-workload-certs" pod "f2407a56-45e8-4257-b271-437daa98e3f4" (UID: "f2407a56-45e8-4257-b271-437daa98e3f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.466609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.466512 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2407a56-45e8-4257-b271-437daa98e3f4-istiod-ca-cert" pod "f2407a56-45e8-4257-b271-437daa98e3f4" (UID: "f2407a56-45e8-4257-b271-437daa98e3f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:01.466879 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.466815 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-data\") pod \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " Oct 30 12:25:01.467044 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.466993 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-workload-socket\") pod \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " Oct 30 12:25:01.467214 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.467164 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2407a56-45e8-4257-b271-437daa98e3f4-istio-token\" (UniqueName: \"kubernetes.io/projected/f2407a56-45e8-4257-b271-437daa98e3f4-istio-token\") pod \"f2407a56-45e8-4257-b271-437daa98e3f4\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " Oct 30 12:25:01.467373 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.467300 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-token\" (UniqueName: \"kubernetes.io/projected/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-token\") pod \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " Oct 30 12:25:01.467373 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.467343 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istiod-ca-cert\") pod \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " Oct 30 12:25:01.467546 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.467396 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-workload-socket" pod "f2407a56-45e8-4257-b271-437daa98e3f4" (UID: "f2407a56-45e8-4257-b271-437daa98e3f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.467773 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.467742 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-credential-socket\") pod \"f2407a56-45e8-4257-b271-437daa98e3f4\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " Oct 30 12:25:01.468015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.467959 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2407a56-45e8-4257-b271-437daa98e3f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2407a56-45e8-4257-b271-437daa98e3f4-istio-podinfo\") pod \"f2407a56-45e8-4257-b271-437daa98e3f4\" (UID: \"f2407a56-45e8-4257-b271-437daa98e3f4\") " Oct 30 12:25:01.468220 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.468170 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-workload-certs\") pod \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " Oct 30 12:25:01.468383 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.468335 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-kube-api-access-rrq2c\" (UniqueName: \"kubernetes.io/projected/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-kube-api-access-rrq2c\") pod \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " Oct 30 12:25:01.468619 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.468575 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-podinfo\") pod \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\" (UID: \"f01f9f03-1c1c-4b0a-bed3-5920537c7ea3\") " Oct 30 12:25:01.470027 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.469972 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.470027 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.469999 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2407a56-45e8-4257-b271-437daa98e3f4-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.470027 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.470014 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.470507 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.470284 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.470507 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.470050 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istiod-ca-cert" pod "f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" (UID: "f01f9f03-1c1c-4b0a-bed3-5920537c7ea3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:01.470507 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.470419 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-istio-envoy" pod "f2407a56-45e8-4257-b271-437daa98e3f4" (UID: "f2407a56-45e8-4257-b271-437daa98e3f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.470725 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.470644 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-credential-socket" pod "f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" (UID: "f01f9f03-1c1c-4b0a-bed3-5920537c7ea3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.471098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.470833 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-credential-socket" pod "f2407a56-45e8-4257-b271-437daa98e3f4" (UID: "f2407a56-45e8-4257-b271-437daa98e3f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.471222 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.471193 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-workload-certs" pod "f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" (UID: "f01f9f03-1c1c-4b0a-bed3-5920537c7ea3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.471498 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.471409 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-workload-socket" pod "f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" (UID: "f01f9f03-1c1c-4b0a-bed3-5920537c7ea3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.471839 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.471598 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-data" pod "f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" (UID: "f01f9f03-1c1c-4b0a-bed3-5920537c7ea3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.473977 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.473951 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:25:01.474122 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.473949 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2407a56-45e8-4257-b271-437daa98e3f4-kube-api-access-2vkdx" pod "f2407a56-45e8-4257-b271-437daa98e3f4" (UID: "f2407a56-45e8-4257-b271-437daa98e3f4"). InnerVolumeSpecName "kube-api-access-2vkdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:01.474937 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.474860 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2407a56-45e8-4257-b271-437daa98e3f4-istio-token" pod "f2407a56-45e8-4257-b271-437daa98e3f4" (UID: "f2407a56-45e8-4257-b271-437daa98e3f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:01.475443 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.475419 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-kube-api-access-rrq2c" pod "f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" (UID: "f01f9f03-1c1c-4b0a-bed3-5920537c7ea3"). InnerVolumeSpecName "kube-api-access-rrq2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:01.476107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.476049 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-envoy" pod "f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" (UID: "f01f9f03-1c1c-4b0a-bed3-5920537c7ea3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.476326 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.476306 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-token" pod "f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" (UID: "f01f9f03-1c1c-4b0a-bed3-5920537c7ea3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:01.477426 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.477401 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2407a56-45e8-4257-b271-437daa98e3f4-istio-podinfo" pod "f2407a56-45e8-4257-b271-437daa98e3f4" (UID: "f2407a56-45e8-4257-b271-437daa98e3f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:25:01.477523 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.477502 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-podinfo" pod "f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" (UID: "f01f9f03-1c1c-4b0a-bed3-5920537c7ea3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:25:01.482718 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.482669 4468 scope.go:122] "RemoveContainer" containerID="b83f019a39ed73320794428933bd63fe967227d54f193e0d6afafa8683d82146" Oct 30 12:25:01.483133 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:25:01.483099 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b83f019a39ed73320794428933bd63fe967227d54f193e0d6afafa8683d82146\": container with ID starting with b83f019a39ed73320794428933bd63fe967227d54f193e0d6afafa8683d82146 not found: ID does not exist" containerID="b83f019a39ed73320794428933bd63fe967227d54f193e0d6afafa8683d82146" Oct 30 12:25:01.483234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.483144 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b83f019a39ed73320794428933bd63fe967227d54f193e0d6afafa8683d82146"} err="failed to get container status \"b83f019a39ed73320794428933bd63fe967227d54f193e0d6afafa8683d82146\": rpc error: code = NotFound desc = could not find container \"b83f019a39ed73320794428933bd63fe967227d54f193e0d6afafa8683d82146\": container with ID starting with b83f019a39ed73320794428933bd63fe967227d54f193e0d6afafa8683d82146 not found: ID does not exist" Oct 30 12:25:01.520171 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.520141 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:25:01.571291 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.571262 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-token\" (UniqueName: \"kubernetes.io/projected/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-token\") pod \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " Oct 30 12:25:01.571459 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.571338 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-envoy\") pod \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " Oct 30 12:25:01.571713 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.571691 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-data\") pod \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " Oct 30 12:25:01.571881 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.571864 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-workload-socket\") pod \"32eeefc6-4a6c-439a-b825-fc318449dc66\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " Oct 30 12:25:01.571994 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.571928 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-data" pod "29b6677b-eb87-4662-9b1b-2c0a4832d1d5" (UID: "29b6677b-eb87-4662-9b1b-2c0a4832d1d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.571994 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.571951 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-kube-api-access-7qv6z\" (UniqueName: \"kubernetes.io/projected/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-kube-api-access-7qv6z\") pod \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " Oct 30 12:25:01.572166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.572017 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-data\") pod \"32eeefc6-4a6c-439a-b825-fc318449dc66\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " Oct 30 12:25:01.572245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.572181 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-credential-socket\") pod \"32eeefc6-4a6c-439a-b825-fc318449dc66\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " Oct 30 12:25:01.572245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.572184 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-workload-socket" pod "32eeefc6-4a6c-439a-b825-fc318449dc66" (UID: "32eeefc6-4a6c-439a-b825-fc318449dc66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.572351 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.572255 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-token\" (UniqueName: \"kubernetes.io/projected/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-token\") pod \"32eeefc6-4a6c-439a-b825-fc318449dc66\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " Oct 30 12:25:01.572415 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.572401 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-workload-certs\") pod \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " Oct 30 12:25:01.572480 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.572460 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-podinfo\") pod \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " Oct 30 12:25:01.572575 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.572553 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-credential-socket" pod "32eeefc6-4a6c-439a-b825-fc318449dc66" (UID: "32eeefc6-4a6c-439a-b825-fc318449dc66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.572864 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.572839 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-data" pod "32eeefc6-4a6c-439a-b825-fc318449dc66" (UID: "32eeefc6-4a6c-439a-b825-fc318449dc66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.573117 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.573085 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-workload-certs" pod "29b6677b-eb87-4662-9b1b-2c0a4832d1d5" (UID: "29b6677b-eb87-4662-9b1b-2c0a4832d1d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.573187 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.573129 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-credential-socket\") pod \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " Oct 30 12:25:01.573187 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.573161 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32eeefc6-4a6c-439a-b825-fc318449dc66-kube-api-access-x55ql\" (UniqueName: \"kubernetes.io/projected/32eeefc6-4a6c-439a-b825-fc318449dc66-kube-api-access-x55ql\") pod \"32eeefc6-4a6c-439a-b825-fc318449dc66\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " Oct 30 12:25:01.573262 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.573218 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-workload-socket\") pod \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " Oct 30 12:25:01.573316 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.573301 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-podinfo\") pod \"32eeefc6-4a6c-439a-b825-fc318449dc66\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " Oct 30 12:25:01.573381 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.573367 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32eeefc6-4a6c-439a-b825-fc318449dc66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32eeefc6-4a6c-439a-b825-fc318449dc66-istiod-ca-cert\") pod \"32eeefc6-4a6c-439a-b825-fc318449dc66\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " Oct 30 12:25:01.573419 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.573409 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istiod-ca-cert\") pod \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\" (UID: \"29b6677b-eb87-4662-9b1b-2c0a4832d1d5\") " Oct 30 12:25:01.573459 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.573445 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-envoy\") pod \"32eeefc6-4a6c-439a-b825-fc318449dc66\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " Oct 30 12:25:01.573494 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.573484 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-workload-certs\") pod \"32eeefc6-4a6c-439a-b825-fc318449dc66\" (UID: \"32eeefc6-4a6c-439a-b825-fc318449dc66\") " Oct 30 12:25:01.574912 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.574883 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.575030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.574916 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.575030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.574931 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.575030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.574944 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.575030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.574957 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2407a56-45e8-4257-b271-437daa98e3f4-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.575030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.574971 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.575030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.574984 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.575030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.574996 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.575030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.575009 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2407a56-45e8-4257-b271-437daa98e3f4-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.575030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.575023 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.575030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.575035 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.575718 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.575048 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.575718 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.575062 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrq2c\" (UniqueName: \"kubernetes.io/projected/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-kube-api-access-rrq2c\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.575718 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.575094 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.575718 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.575135 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.575718 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.575148 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.575718 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.575163 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.575718 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.575178 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vkdx\" (UniqueName: \"kubernetes.io/projected/f2407a56-45e8-4257-b271-437daa98e3f4-kube-api-access-2vkdx\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.575718 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.575192 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2407a56-45e8-4257-b271-437daa98e3f4-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.575718 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.575387 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-workload-certs" pod "32eeefc6-4a6c-439a-b825-fc318449dc66" (UID: "32eeefc6-4a6c-439a-b825-fc318449dc66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.576373 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.575869 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32eeefc6-4a6c-439a-b825-fc318449dc66-istiod-ca-cert" pod "32eeefc6-4a6c-439a-b825-fc318449dc66" (UID: "32eeefc6-4a6c-439a-b825-fc318449dc66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:01.576373 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.576220 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istiod-ca-cert" pod "29b6677b-eb87-4662-9b1b-2c0a4832d1d5" (UID: "29b6677b-eb87-4662-9b1b-2c0a4832d1d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:01.576373 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.576254 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-token" pod "32eeefc6-4a6c-439a-b825-fc318449dc66" (UID: "32eeefc6-4a6c-439a-b825-fc318449dc66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:01.576544 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.576400 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-podinfo" pod "29b6677b-eb87-4662-9b1b-2c0a4832d1d5" (UID: "29b6677b-eb87-4662-9b1b-2c0a4832d1d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:25:01.576544 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.576494 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz"] Oct 30 12:25:01.576654 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.576636 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-podinfo" pod "32eeefc6-4a6c-439a-b825-fc318449dc66" (UID: "32eeefc6-4a6c-439a-b825-fc318449dc66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:25:01.576739 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.576714 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-credential-socket" pod "29b6677b-eb87-4662-9b1b-2c0a4832d1d5" (UID: "29b6677b-eb87-4662-9b1b-2c0a4832d1d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.576796 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.576725 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-workload-socket" pod "29b6677b-eb87-4662-9b1b-2c0a4832d1d5" (UID: "29b6677b-eb87-4662-9b1b-2c0a4832d1d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.580311 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.580287 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-envoy" pod "32eeefc6-4a6c-439a-b825-fc318449dc66" (UID: "32eeefc6-4a6c-439a-b825-fc318449dc66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.580414 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:25:01.580316 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a13adda_0849_4688_a331_04b6701a8325.slice/crio-d412d89c1d67fcdc256adb1416639c5e9f83451f5ebceb9165b00f4ea3f011a0 WatchSource:0}: Error finding container d412d89c1d67fcdc256adb1416639c5e9f83451f5ebceb9165b00f4ea3f011a0: Status 404 returned error can't find the container with id d412d89c1d67fcdc256adb1416639c5e9f83451f5ebceb9165b00f4ea3f011a0 Oct 30 12:25:01.580479 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.580453 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32eeefc6-4a6c-439a-b825-fc318449dc66-kube-api-access-x55ql" pod "32eeefc6-4a6c-439a-b825-fc318449dc66" (UID: "32eeefc6-4a6c-439a-b825-fc318449dc66"). InnerVolumeSpecName "kube-api-access-x55ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:01.587620 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.587543 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-token" pod "29b6677b-eb87-4662-9b1b-2c0a4832d1d5" (UID: "29b6677b-eb87-4662-9b1b-2c0a4832d1d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:01.590233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.590201 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-kube-api-access-7qv6z" pod "29b6677b-eb87-4662-9b1b-2c0a4832d1d5" (UID: "29b6677b-eb87-4662-9b1b-2c0a4832d1d5"). InnerVolumeSpecName "kube-api-access-7qv6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:01.590336 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.590207 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-envoy" pod "29b6677b-eb87-4662-9b1b-2c0a4832d1d5" (UID: "29b6677b-eb87-4662-9b1b-2c0a4832d1d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:01.676496 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.676452 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.676496 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.676496 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x55ql\" (UniqueName: \"kubernetes.io/projected/32eeefc6-4a6c-439a-b825-fc318449dc66-kube-api-access-x55ql\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.676946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.676507 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.676946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.676516 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.676946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.676525 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32eeefc6-4a6c-439a-b825-fc318449dc66-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.676946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.676532 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.676946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.676540 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.676946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.676548 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32eeefc6-4a6c-439a-b825-fc318449dc66-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.676946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.676559 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.676946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.676570 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.676946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.676583 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qv6z\" (UniqueName: \"kubernetes.io/projected/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-kube-api-access-7qv6z\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.676946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.676595 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32eeefc6-4a6c-439a-b825-fc318449dc66-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.676946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.676608 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b6677b-eb87-4662-9b1b-2c0a4832d1d5-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:01.745015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.744967 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw"] Oct 30 12:25:01.750274 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.750193 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-48bsw"] Oct 30 12:25:01.831702 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.831652 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z"] Oct 30 12:25:01.843915 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:01.843841 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-w897z"] Oct 30 12:25:02.405145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:02.405107 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz" event={"ID":"6a13adda-0849-4688-a331-04b6701a8325","Type":"ContainerStarted","Data":"1645fa0ca63621780d7458b320b5301d8e47febc0c9aaadf7f8462e4c58d233a"} Oct 30 12:25:02.405608 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:02.405166 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz" event={"ID":"6a13adda-0849-4688-a331-04b6701a8325","Type":"ContainerStarted","Data":"d412d89c1d67fcdc256adb1416639c5e9f83451f5ebceb9165b00f4ea3f011a0"} Oct 30 12:25:02.405608 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:02.405202 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz" Oct 30 12:25:02.406438 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:02.406405 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" Oct 30 12:25:02.406438 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:02.406426 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6" event={"ID":"29b6677b-eb87-4662-9b1b-2c0a4832d1d5","Type":"ContainerDied","Data":"13496f895be400ea4178652cf12b2206c029494773c4f69288b817c99100eb59"} Oct 30 12:25:02.406636 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:02.406479 4468 scope.go:122] "RemoveContainer" containerID="394b075ee687c46161d0f994ab4d474aff633fceb5cc2d632bc009192ba2560a" Oct 30 12:25:02.408159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:02.408122 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn" event={"ID":"82ada6d9-8378-48b5-aac2-38294aa6ef5c","Type":"ContainerStarted","Data":"0aa1853dbf536b8cfad6a04605fa0f4b9e380dab3ee0cc16b008f09e9f642c07"} Oct 30 12:25:02.408267 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:02.408172 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn" event={"ID":"82ada6d9-8378-48b5-aac2-38294aa6ef5c","Type":"ContainerStarted","Data":"d68bcb98cbbf64af34f5bc7657d7e024183f9972431c8c6ea8e54d641b7b8e31"} Oct 30 12:25:02.408267 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:02.408203 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn" Oct 30 12:25:02.412081 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:02.412046 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" Oct 30 12:25:02.412203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:02.412106 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn" event={"ID":"32eeefc6-4a6c-439a-b825-fc318449dc66","Type":"ContainerDied","Data":"b49e61ac0593b487d78631ff1e27591e415e52ac8c200478a12767a453150ce7"} Oct 30 12:25:02.419217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:02.419062 4468 scope.go:122] "RemoveContainer" containerID="4b472ff45a71f0849d987df82a0a77be300887d40a7996850261cb7a14a1ae84" Oct 30 12:25:02.452639 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:02.452582 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz" podStartSLOduration=2.452564806 podStartE2EDuration="2.452564806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:02.447681705 +0000 UTC m=+1853.481820101" watchObservedRunningTime="2026-10-30 12:25:02.452564806 +0000 UTC m=+1853.486703201" Oct 30 12:25:02.494037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:02.493981 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn" podStartSLOduration=2.49396632 podStartE2EDuration="2.49396632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:02.490051512 +0000 UTC m=+1853.524189908" watchObservedRunningTime="2026-10-30 12:25:02.49396632 +0000 UTC m=+1853.528104988" Oct 30 12:25:02.532832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:02.532795 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6"] Oct 30 12:25:02.546122 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:02.546044 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-s6qc6"] Oct 30 12:25:02.586603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:02.586555 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn"] Oct 30 12:25:02.591230 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:02.591202 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-t5xwn"] Oct 30 12:25:03.101680 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.101646 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b6677b-eb87-4662-9b1b-2c0a4832d1d5" path="/var/lib/kubelet/pods/29b6677b-eb87-4662-9b1b-2c0a4832d1d5/volumes" Oct 30 12:25:03.102209 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.102188 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32eeefc6-4a6c-439a-b825-fc318449dc66" path="/var/lib/kubelet/pods/32eeefc6-4a6c-439a-b825-fc318449dc66/volumes" Oct 30 12:25:03.102645 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.102628 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" path="/var/lib/kubelet/pods/f01f9f03-1c1c-4b0a-bed3-5920537c7ea3/volumes" Oct 30 12:25:03.103433 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.103416 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2407a56-45e8-4257-b271-437daa98e3f4" path="/var/lib/kubelet/pods/f2407a56-45e8-4257-b271-437daa98e3f4/volumes" Oct 30 12:25:03.716743 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.716710 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj"] Oct 30 12:25:03.717383 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.717366 4468 cpu_manager.go:414] "containerMap: removing container" podUID="f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" containerName="istio-proxy" Oct 30 12:25:03.717428 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.717383 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" containerName="istio-proxy" Oct 30 12:25:03.717428 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.717405 4468 cpu_manager.go:414] "containerMap: removing container" podUID="29b6677b-eb87-4662-9b1b-2c0a4832d1d5" containerName="istio-proxy" Oct 30 12:25:03.717428 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.717410 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b6677b-eb87-4662-9b1b-2c0a4832d1d5" containerName="istio-proxy" Oct 30 12:25:03.717522 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.717429 4468 cpu_manager.go:414] "containerMap: removing container" podUID="f2407a56-45e8-4257-b271-437daa98e3f4" containerName="istio-proxy" Oct 30 12:25:03.717522 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.717435 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2407a56-45e8-4257-b271-437daa98e3f4" containerName="istio-proxy" Oct 30 12:25:03.717522 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.717456 4468 cpu_manager.go:414] "containerMap: removing container" podUID="32eeefc6-4a6c-439a-b825-fc318449dc66" containerName="istio-proxy" Oct 30 12:25:03.717522 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.717460 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32eeefc6-4a6c-439a-b825-fc318449dc66" containerName="istio-proxy" Oct 30 12:25:03.717654 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.717554 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="32eeefc6-4a6c-439a-b825-fc318449dc66" containerName="istio-proxy" Oct 30 12:25:03.717654 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.717562 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01f9f03-1c1c-4b0a-bed3-5920537c7ea3" containerName="istio-proxy" Oct 30 12:25:03.717654 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.717571 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b6677b-eb87-4662-9b1b-2c0a4832d1d5" containerName="istio-proxy" Oct 30 12:25:03.717654 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.717579 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2407a56-45e8-4257-b271-437daa98e3f4" containerName="istio-proxy" Oct 30 12:25:03.722615 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.722599 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" Oct 30 12:25:03.743632 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.743592 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj"] Oct 30 12:25:03.756434 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.756394 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-aef61e-serving-cert\"" Oct 30 12:25:03.812876 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.812835 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aed0019-79e9-4471-97b9-830763499da0-openshift-service-ca\") pod \"maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj\" (UID: \"6aed0019-79e9-4471-97b9-830763499da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" Oct 30 12:25:03.813051 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.812956 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h62qf\" (UniqueName: \"kubernetes.io/projected/6aed0019-79e9-4471-97b9-830763499da0-kube-api-access-h62qf\") pod \"maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj\" (UID: \"6aed0019-79e9-4471-97b9-830763499da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" Oct 30 12:25:03.813148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.813050 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aed0019-79e9-4471-97b9-830763499da0-metrics-certs\") pod \"maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj\" (UID: \"6aed0019-79e9-4471-97b9-830763499da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" Oct 30 12:25:03.813202 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.813187 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6aed0019-79e9-4471-97b9-830763499da0-maas-api-tls\") pod \"maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj\" (UID: \"6aed0019-79e9-4471-97b9-830763499da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" Oct 30 12:25:03.927097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.926397 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aed0019-79e9-4471-97b9-830763499da0-metrics-certs\") pod \"maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj\" (UID: \"6aed0019-79e9-4471-97b9-830763499da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" Oct 30 12:25:03.927097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.926516 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6aed0019-79e9-4471-97b9-830763499da0-maas-api-tls\") pod \"maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj\" (UID: \"6aed0019-79e9-4471-97b9-830763499da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" Oct 30 12:25:03.927097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.926666 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aed0019-79e9-4471-97b9-830763499da0-openshift-service-ca\") pod \"maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj\" (UID: \"6aed0019-79e9-4471-97b9-830763499da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" Oct 30 12:25:03.927097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.926745 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h62qf\" (UniqueName: \"kubernetes.io/projected/6aed0019-79e9-4471-97b9-830763499da0-kube-api-access-h62qf\") pod \"maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj\" (UID: \"6aed0019-79e9-4471-97b9-830763499da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" Oct 30 12:25:03.951150 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.949010 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aed0019-79e9-4471-97b9-830763499da0-openshift-service-ca\") pod \"maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj\" (UID: \"6aed0019-79e9-4471-97b9-830763499da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" Oct 30 12:25:03.973387 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.962883 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6aed0019-79e9-4471-97b9-830763499da0-maas-api-tls\") pod \"maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj\" (UID: \"6aed0019-79e9-4471-97b9-830763499da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" Oct 30 12:25:03.973694 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.966732 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h62qf\" (UniqueName: \"kubernetes.io/projected/6aed0019-79e9-4471-97b9-830763499da0-kube-api-access-h62qf\") pod \"maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj\" (UID: \"6aed0019-79e9-4471-97b9-830763499da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" Oct 30 12:25:03.973694 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:03.968594 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aed0019-79e9-4471-97b9-830763499da0-metrics-certs\") pod \"maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj\" (UID: \"6aed0019-79e9-4471-97b9-830763499da0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" Oct 30 12:25:04.049163 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:04.045344 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" Oct 30 12:25:04.362441 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:25:04.362403 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aed0019_79e9_4471_97b9_830763499da0.slice/crio-0868f8ec9f6ab4e26460f53ff0de035ecbea4686119e4cb41f33afc6782ec9c9 WatchSource:0}: Error finding container 0868f8ec9f6ab4e26460f53ff0de035ecbea4686119e4cb41f33afc6782ec9c9: Status 404 returned error can't find the container with id 0868f8ec9f6ab4e26460f53ff0de035ecbea4686119e4cb41f33afc6782ec9c9 Oct 30 12:25:04.364220 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:04.364195 4468 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:25:04.364571 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:04.364539 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj"] Oct 30 12:25:04.430632 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:04.430597 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" event={"ID":"6aed0019-79e9-4471-97b9-830763499da0","Type":"ContainerStarted","Data":"0868f8ec9f6ab4e26460f53ff0de035ecbea4686119e4cb41f33afc6782ec9c9"} Oct 30 12:25:04.877767 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:04.877732 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47"] Oct 30 12:25:04.882181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:04.882157 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47" Oct 30 12:25:04.905347 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:04.905314 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47"] Oct 30 12:25:04.912515 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:04.912479 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8e3112-dockercfg-2v77n\"" Oct 30 12:25:04.912515 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:04.912504 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8e3112\"" Oct 30 12:25:04.950197 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:04.950153 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9868d552-736b-4752-9272-8805b53a8dcf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47\" (UID: \"9868d552-736b-4752-9272-8805b53a8dcf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47" Oct 30 12:25:04.950524 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:04.950494 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pcs6\" (UniqueName: \"kubernetes.io/projected/9868d552-736b-4752-9272-8805b53a8dcf-kube-api-access-7pcs6\") pod \"payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47\" (UID: \"9868d552-736b-4752-9272-8805b53a8dcf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47" Oct 30 12:25:04.958131 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:04.958085 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh"] Oct 30 12:25:04.962985 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:04.962950 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh" Oct 30 12:25:05.016589 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:05.016500 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh"] Oct 30 12:25:05.124095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:05.112696 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60785db9-4e26-4614-ab7c-490407d0095b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh\" (UID: \"60785db9-4e26-4614-ab7c-490407d0095b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh" Oct 30 12:25:05.159098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:05.145768 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9868d552-736b-4752-9272-8805b53a8dcf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47\" (UID: \"9868d552-736b-4752-9272-8805b53a8dcf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47" Oct 30 12:25:05.159098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:05.146026 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcs6\" (UniqueName: \"kubernetes.io/projected/9868d552-736b-4752-9272-8805b53a8dcf-kube-api-access-7pcs6\") pod \"payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47\" (UID: \"9868d552-736b-4752-9272-8805b53a8dcf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47" Oct 30 12:25:05.159098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:05.146121 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztcwk\" (UniqueName: \"kubernetes.io/projected/60785db9-4e26-4614-ab7c-490407d0095b-kube-api-access-ztcwk\") pod \"payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh\" (UID: \"60785db9-4e26-4614-ab7c-490407d0095b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh" Oct 30 12:25:05.159098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:05.147497 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9868d552-736b-4752-9272-8805b53a8dcf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47\" (UID: \"9868d552-736b-4752-9272-8805b53a8dcf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47" Oct 30 12:25:05.221833 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:05.221802 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pcs6\" (UniqueName: \"kubernetes.io/projected/9868d552-736b-4752-9272-8805b53a8dcf-kube-api-access-7pcs6\") pod \"payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47\" (UID: \"9868d552-736b-4752-9272-8805b53a8dcf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47" Oct 30 12:25:05.357454 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:05.355010 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb" Oct 30 12:25:05.357454 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:05.357255 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60785db9-4e26-4614-ab7c-490407d0095b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh\" (UID: \"60785db9-4e26-4614-ab7c-490407d0095b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh" Oct 30 12:25:05.357685 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:05.357576 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztcwk\" (UniqueName: \"kubernetes.io/projected/60785db9-4e26-4614-ab7c-490407d0095b-kube-api-access-ztcwk\") pod \"payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh\" (UID: \"60785db9-4e26-4614-ab7c-490407d0095b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh" Oct 30 12:25:05.359390 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:05.359355 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60785db9-4e26-4614-ab7c-490407d0095b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh\" (UID: \"60785db9-4e26-4614-ab7c-490407d0095b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh" Oct 30 12:25:05.385845 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:05.385810 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztcwk\" (UniqueName: \"kubernetes.io/projected/60785db9-4e26-4614-ab7c-490407d0095b-kube-api-access-ztcwk\") pod \"payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh\" (UID: \"60785db9-4e26-4614-ab7c-490407d0095b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh" Oct 30 12:25:05.445978 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:05.445937 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" event={"ID":"6aed0019-79e9-4471-97b9-830763499da0","Type":"ContainerStarted","Data":"73dc006e959a8f5aea45ab4b7fc38f8d0e7823acb98142adb09fff073879ce62"} Oct 30 12:25:05.446188 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:05.446045 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" Oct 30 12:25:05.494313 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:05.494260 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" podStartSLOduration=2.130171082 podStartE2EDuration="2.494243269s" totalImagesPullingTime="364.072187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:05.493694514 +0000 UTC m=+1856.527832909" watchObservedRunningTime="2026-10-30 12:25:05.494243269 +0000 UTC m=+1856.528381665" Oct 30 12:25:05.496995 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:05.496966 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47" Oct 30 12:25:05.582904 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:05.582868 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh" Oct 30 12:25:05.818119 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:25:05.818084 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9868d552_736b_4752_9272_8805b53a8dcf.slice/crio-3a248e032dac8ebc78b415ee0e2d09b2d13a7ce678588a16abb74c7ff02b72df WatchSource:0}: Error finding container 3a248e032dac8ebc78b415ee0e2d09b2d13a7ce678588a16abb74c7ff02b72df: Status 404 returned error can't find the container with id 3a248e032dac8ebc78b415ee0e2d09b2d13a7ce678588a16abb74c7ff02b72df Oct 30 12:25:05.822699 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:05.822624 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47"] Oct 30 12:25:05.964148 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:25:05.964056 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60785db9_4e26_4614_ab7c_490407d0095b.slice/crio-dbf4315dddd7781bd855d48aeef9c4306e82950789a6780f86b8be58dae9cb5e WatchSource:0}: Error finding container dbf4315dddd7781bd855d48aeef9c4306e82950789a6780f86b8be58dae9cb5e: Status 404 returned error can't find the container with id dbf4315dddd7781bd855d48aeef9c4306e82950789a6780f86b8be58dae9cb5e Oct 30 12:25:05.968978 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:05.968942 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh"] Oct 30 12:25:06.216474 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:06.216404 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv" Oct 30 12:25:06.456161 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:06.456113 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh" event={"ID":"60785db9-4e26-4614-ab7c-490407d0095b","Type":"ContainerStarted","Data":"5ddded07342ba731829dac140c82c1e15bd4fd671f54b632bb14de53de00e186"} Oct 30 12:25:06.456385 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:06.456190 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh" event={"ID":"60785db9-4e26-4614-ab7c-490407d0095b","Type":"ContainerStarted","Data":"dbf4315dddd7781bd855d48aeef9c4306e82950789a6780f86b8be58dae9cb5e"} Oct 30 12:25:06.456385 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:06.456220 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh" Oct 30 12:25:06.458039 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:06.458007 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47" event={"ID":"9868d552-736b-4752-9272-8805b53a8dcf","Type":"ContainerStarted","Data":"05b1a9734ac39896e81c728cd91aac4f86c00b0e727d1e9b3b1bb7bffbcfc935"} Oct 30 12:25:06.458209 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:06.458106 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47" event={"ID":"9868d552-736b-4752-9272-8805b53a8dcf","Type":"ContainerStarted","Data":"3a248e032dac8ebc78b415ee0e2d09b2d13a7ce678588a16abb74c7ff02b72df"} Oct 30 12:25:06.458209 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:06.458142 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47" Oct 30 12:25:06.502723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:06.502623 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh" podStartSLOduration=2.502609434 podStartE2EDuration="2.502609434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:06.498979384 +0000 UTC m=+1857.533117779" watchObservedRunningTime="2026-10-30 12:25:06.502609434 +0000 UTC m=+1857.536747829" Oct 30 12:25:06.549349 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:06.549292 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47" podStartSLOduration=2.549277103 podStartE2EDuration="2.549277103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:06.544970317 +0000 UTC m=+1857.579108714" watchObservedRunningTime="2026-10-30 12:25:06.549277103 +0000 UTC m=+1857.583415496" Oct 30 12:25:11.398998 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:11.398966 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6" Oct 30 12:25:11.400974 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:11.400952 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx" Oct 30 12:25:11.466285 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:11.466255 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" Oct 30 12:25:13.425600 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:13.425567 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn" Oct 30 12:25:13.426124 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:13.425635 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz" Oct 30 12:25:13.886272 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:13.886168 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb"] Oct 30 12:25:14.459906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:14.459871 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv"] Oct 30 12:25:14.466682 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:14.466651 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" Oct 30 12:25:14.488980 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:14.488944 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv"] Oct 30 12:25:14.499715 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:14.499685 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-876549-serving-cert\"" Oct 30 12:25:14.611742 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:14.611710 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e5e17a1-d0c2-4357-827e-0b83a118f510-metrics-certs\") pod \"maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv\" (UID: \"4e5e17a1-d0c2-4357-827e-0b83a118f510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" Oct 30 12:25:14.611742 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:14.611747 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bp7k\" (UniqueName: \"kubernetes.io/projected/4e5e17a1-d0c2-4357-827e-0b83a118f510-kube-api-access-5bp7k\") pod \"maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv\" (UID: \"4e5e17a1-d0c2-4357-827e-0b83a118f510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" Oct 30 12:25:14.612009 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:14.611883 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e5e17a1-d0c2-4357-827e-0b83a118f510-maas-api-tls\") pod \"maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv\" (UID: \"4e5e17a1-d0c2-4357-827e-0b83a118f510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" Oct 30 12:25:14.612100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:14.612029 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5e17a1-d0c2-4357-827e-0b83a118f510-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv\" (UID: \"4e5e17a1-d0c2-4357-827e-0b83a118f510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" Oct 30 12:25:14.726207 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:14.720586 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e5e17a1-d0c2-4357-827e-0b83a118f510-metrics-certs\") pod \"maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv\" (UID: \"4e5e17a1-d0c2-4357-827e-0b83a118f510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" Oct 30 12:25:14.726207 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:14.720642 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bp7k\" (UniqueName: \"kubernetes.io/projected/4e5e17a1-d0c2-4357-827e-0b83a118f510-kube-api-access-5bp7k\") pod \"maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv\" (UID: \"4e5e17a1-d0c2-4357-827e-0b83a118f510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" Oct 30 12:25:14.726207 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:14.720725 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e5e17a1-d0c2-4357-827e-0b83a118f510-maas-api-tls\") pod \"maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv\" (UID: \"4e5e17a1-d0c2-4357-827e-0b83a118f510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" Oct 30 12:25:14.726207 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:14.720832 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5e17a1-d0c2-4357-827e-0b83a118f510-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv\" (UID: \"4e5e17a1-d0c2-4357-827e-0b83a118f510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" Oct 30 12:25:14.726207 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:14.721798 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5e17a1-d0c2-4357-827e-0b83a118f510-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv\" (UID: \"4e5e17a1-d0c2-4357-827e-0b83a118f510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" Oct 30 12:25:14.732340 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:14.732307 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e5e17a1-d0c2-4357-827e-0b83a118f510-maas-api-tls\") pod \"maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv\" (UID: \"4e5e17a1-d0c2-4357-827e-0b83a118f510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" Oct 30 12:25:14.747144 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:14.736628 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e5e17a1-d0c2-4357-827e-0b83a118f510-metrics-certs\") pod \"maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv\" (UID: \"4e5e17a1-d0c2-4357-827e-0b83a118f510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" Oct 30 12:25:14.788953 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:14.788848 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bp7k\" (UniqueName: \"kubernetes.io/projected/4e5e17a1-d0c2-4357-827e-0b83a118f510-kube-api-access-5bp7k\") pod \"maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv\" (UID: \"4e5e17a1-d0c2-4357-827e-0b83a118f510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" Oct 30 12:25:14.816904 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:14.816856 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" Oct 30 12:25:15.220544 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:25:15.220514 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5e17a1_d0c2_4357_827e_0b83a118f510.slice/crio-a26086da2d1a4bf3441730ebfc1d74e238f333e4203c92ef5566f7a802ee5a60 WatchSource:0}: Error finding container a26086da2d1a4bf3441730ebfc1d74e238f333e4203c92ef5566f7a802ee5a60: Status 404 returned error can't find the container with id a26086da2d1a4bf3441730ebfc1d74e238f333e4203c92ef5566f7a802ee5a60 Oct 30 12:25:15.226458 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:15.225063 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv"] Oct 30 12:25:15.539543 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:15.539503 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" event={"ID":"4e5e17a1-d0c2-4357-827e-0b83a118f510","Type":"ContainerStarted","Data":"a26086da2d1a4bf3441730ebfc1d74e238f333e4203c92ef5566f7a802ee5a60"} Oct 30 12:25:16.349040 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.348996 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2"] Oct 30 12:25:16.353700 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.353672 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" Oct 30 12:25:16.375953 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.375914 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2"] Oct 30 12:25:16.381399 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.381368 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7e2db2-serving-cert\"" Oct 30 12:25:16.506823 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.506783 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90b0b79c-9489-4285-a629-aaf506cbecdc-metrics-certs\") pod \"maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2\" (UID: \"90b0b79c-9489-4285-a629-aaf506cbecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" Oct 30 12:25:16.507008 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.506957 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90b0b79c-9489-4285-a629-aaf506cbecdc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2\" (UID: \"90b0b79c-9489-4285-a629-aaf506cbecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" Oct 30 12:25:16.507099 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.507027 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mhm7\" (UniqueName: \"kubernetes.io/projected/90b0b79c-9489-4285-a629-aaf506cbecdc-kube-api-access-2mhm7\") pod \"maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2\" (UID: \"90b0b79c-9489-4285-a629-aaf506cbecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" Oct 30 12:25:16.507243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.507099 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90b0b79c-9489-4285-a629-aaf506cbecdc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2\" (UID: \"90b0b79c-9489-4285-a629-aaf506cbecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" Oct 30 12:25:16.613310 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.613223 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90b0b79c-9489-4285-a629-aaf506cbecdc-metrics-certs\") pod \"maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2\" (UID: \"90b0b79c-9489-4285-a629-aaf506cbecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" Oct 30 12:25:16.614034 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.614007 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90b0b79c-9489-4285-a629-aaf506cbecdc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2\" (UID: \"90b0b79c-9489-4285-a629-aaf506cbecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" Oct 30 12:25:16.614260 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.614232 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mhm7\" (UniqueName: \"kubernetes.io/projected/90b0b79c-9489-4285-a629-aaf506cbecdc-kube-api-access-2mhm7\") pod \"maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2\" (UID: \"90b0b79c-9489-4285-a629-aaf506cbecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" Oct 30 12:25:16.614486 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.614438 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90b0b79c-9489-4285-a629-aaf506cbecdc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2\" (UID: \"90b0b79c-9489-4285-a629-aaf506cbecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" Oct 30 12:25:16.618455 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.617110 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90b0b79c-9489-4285-a629-aaf506cbecdc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2\" (UID: \"90b0b79c-9489-4285-a629-aaf506cbecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" Oct 30 12:25:16.618636 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.618605 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" event={"ID":"4e5e17a1-d0c2-4357-827e-0b83a118f510","Type":"ContainerStarted","Data":"255b0e84a79f3585df2c8f17ee023dedf7f16ee1adf4d0d4c880e9b73a70ff71"} Oct 30 12:25:16.619014 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.618996 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" Oct 30 12:25:16.635708 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.635638 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90b0b79c-9489-4285-a629-aaf506cbecdc-metrics-certs\") pod \"maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2\" (UID: \"90b0b79c-9489-4285-a629-aaf506cbecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" Oct 30 12:25:16.639940 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.639877 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90b0b79c-9489-4285-a629-aaf506cbecdc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2\" (UID: \"90b0b79c-9489-4285-a629-aaf506cbecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" Oct 30 12:25:16.647509 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.647416 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mhm7\" (UniqueName: \"kubernetes.io/projected/90b0b79c-9489-4285-a629-aaf506cbecdc-kube-api-access-2mhm7\") pod \"maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2\" (UID: \"90b0b79c-9489-4285-a629-aaf506cbecdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" Oct 30 12:25:16.665275 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.665214 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" podStartSLOduration=2.26645985 podStartE2EDuration="2.665196785s" totalImagesPullingTime="398.736935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:16.664236773 +0000 UTC m=+1867.698375170" watchObservedRunningTime="2026-10-30 12:25:16.665196785 +0000 UTC m=+1867.699335179" Oct 30 12:25:16.676843 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.676808 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" Oct 30 12:25:16.990193 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:25:16.990158 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b0b79c_9489_4285_a629_aaf506cbecdc.slice/crio-6afb4834e8fb2f273804bba29e2bee7fee7c08f7009c5aded8f3788893fa2da4 WatchSource:0}: Error finding container 6afb4834e8fb2f273804bba29e2bee7fee7c08f7009c5aded8f3788893fa2da4: Status 404 returned error can't find the container with id 6afb4834e8fb2f273804bba29e2bee7fee7c08f7009c5aded8f3788893fa2da4 Oct 30 12:25:16.993582 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:16.993559 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2"] Oct 30 12:25:17.470432 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:17.470385 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh" Oct 30 12:25:17.471782 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:17.471752 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47" Oct 30 12:25:17.628224 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:17.628130 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" event={"ID":"90b0b79c-9489-4285-a629-aaf506cbecdc","Type":"ContainerStarted","Data":"90518be0484f68ecc5ef58610d7b9cd4c7d7bb77e2795e0fbdc78957defb1c00"} Oct 30 12:25:17.628224 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:17.628202 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" event={"ID":"90b0b79c-9489-4285-a629-aaf506cbecdc","Type":"ContainerStarted","Data":"6afb4834e8fb2f273804bba29e2bee7fee7c08f7009c5aded8f3788893fa2da4"} Oct 30 12:25:17.628647 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:17.628419 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" Oct 30 12:25:17.729805 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:17.700142 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" podStartSLOduration=1.331777194 podStartE2EDuration="1.700116455s" totalImagesPullingTime="368.339261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:17.680979457 +0000 UTC m=+1868.715117854" watchObservedRunningTime="2026-10-30 12:25:17.700116455 +0000 UTC m=+1868.734254854" Oct 30 12:25:18.887365 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:18.887323 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb" podUID="fbeb09c8-bb44-43ab-b2f6-44257398a5e2" containerName="payload-pre-processing" containerID="cri-o://5f75676689c6b9e99cfd4875696d0016624bbbcbd446d8c617afa8f888285767" gracePeriod=25 Oct 30 12:25:19.167290 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:19.167251 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb" Oct 30 12:25:19.333919 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:19.333885 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbeb09c8-bb44-43ab-b2f6-44257398a5e2-kube-api-access-qrrrh\" (UniqueName: \"kubernetes.io/projected/fbeb09c8-bb44-43ab-b2f6-44257398a5e2-kube-api-access-qrrrh\") pod \"fbeb09c8-bb44-43ab-b2f6-44257398a5e2\" (UID: \"fbeb09c8-bb44-43ab-b2f6-44257398a5e2\") " Oct 30 12:25:19.334185 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:19.333961 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbeb09c8-bb44-43ab-b2f6-44257398a5e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbeb09c8-bb44-43ab-b2f6-44257398a5e2-plugins-config-volume\") pod \"fbeb09c8-bb44-43ab-b2f6-44257398a5e2\" (UID: \"fbeb09c8-bb44-43ab-b2f6-44257398a5e2\") " Oct 30 12:25:19.334442 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:19.334414 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbeb09c8-bb44-43ab-b2f6-44257398a5e2-plugins-config-volume" pod "fbeb09c8-bb44-43ab-b2f6-44257398a5e2" (UID: "fbeb09c8-bb44-43ab-b2f6-44257398a5e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:19.334623 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:19.334602 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbeb09c8-bb44-43ab-b2f6-44257398a5e2-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:19.336193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:19.336170 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbeb09c8-bb44-43ab-b2f6-44257398a5e2-kube-api-access-qrrrh" pod "fbeb09c8-bb44-43ab-b2f6-44257398a5e2" (UID: "fbeb09c8-bb44-43ab-b2f6-44257398a5e2"). InnerVolumeSpecName "kube-api-access-qrrrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:19.435713 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:19.435674 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrrrh\" (UniqueName: \"kubernetes.io/projected/fbeb09c8-bb44-43ab-b2f6-44257398a5e2-kube-api-access-qrrrh\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:19.643157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:19.643115 4468 generic.go:356] "Generic (PLEG): container finished" podID="fbeb09c8-bb44-43ab-b2f6-44257398a5e2" containerID="5f75676689c6b9e99cfd4875696d0016624bbbcbd446d8c617afa8f888285767" exitCode=0 Oct 30 12:25:19.643157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:19.643170 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb" Oct 30 12:25:19.643369 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:19.643200 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb" event={"ID":"fbeb09c8-bb44-43ab-b2f6-44257398a5e2","Type":"ContainerDied","Data":"5f75676689c6b9e99cfd4875696d0016624bbbcbd446d8c617afa8f888285767"} Oct 30 12:25:19.643369 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:19.643262 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb" event={"ID":"fbeb09c8-bb44-43ab-b2f6-44257398a5e2","Type":"ContainerDied","Data":"ca5361cfc9d8f519025945dcb3c3b3bbcb1e1328a87145dd5f3dfc2f93b39a80"} Oct 30 12:25:19.643369 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:19.643286 4468 scope.go:122] "RemoveContainer" containerID="5f75676689c6b9e99cfd4875696d0016624bbbcbd446d8c617afa8f888285767" Oct 30 12:25:19.655215 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:19.655187 4468 scope.go:122] "RemoveContainer" containerID="5f75676689c6b9e99cfd4875696d0016624bbbcbd446d8c617afa8f888285767" Oct 30 12:25:19.655593 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:25:19.655572 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f75676689c6b9e99cfd4875696d0016624bbbcbd446d8c617afa8f888285767\": container with ID starting with 5f75676689c6b9e99cfd4875696d0016624bbbcbd446d8c617afa8f888285767 not found: ID does not exist" containerID="5f75676689c6b9e99cfd4875696d0016624bbbcbd446d8c617afa8f888285767" Oct 30 12:25:19.655672 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:19.655607 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f75676689c6b9e99cfd4875696d0016624bbbcbd446d8c617afa8f888285767"} err="failed to get container status \"5f75676689c6b9e99cfd4875696d0016624bbbcbd446d8c617afa8f888285767\": rpc error: code = NotFound desc = could not find container \"5f75676689c6b9e99cfd4875696d0016624bbbcbd446d8c617afa8f888285767\": container with ID starting with 5f75676689c6b9e99cfd4875696d0016624bbbcbd446d8c617afa8f888285767 not found: ID does not exist" Oct 30 12:25:19.701627 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:19.701536 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb"] Oct 30 12:25:19.715334 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:19.715301 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-951ad1-559c7ccf8c-mhtlb"] Oct 30 12:25:21.090655 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:21.090618 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbeb09c8-bb44-43ab-b2f6-44257398a5e2" path="/var/lib/kubelet/pods/fbeb09c8-bb44-43ab-b2f6-44257398a5e2/volumes" Oct 30 12:25:22.634137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:22.634102 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" Oct 30 12:25:23.213785 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.213749 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst"] Oct 30 12:25:23.214437 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.214422 4468 cpu_manager.go:414] "containerMap: removing container" podUID="fbeb09c8-bb44-43ab-b2f6-44257398a5e2" containerName="payload-pre-processing" Oct 30 12:25:23.214509 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.214438 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbeb09c8-bb44-43ab-b2f6-44257398a5e2" containerName="payload-pre-processing" Oct 30 12:25:23.214996 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.214560 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbeb09c8-bb44-43ab-b2f6-44257398a5e2" containerName="payload-pre-processing" Oct 30 12:25:23.219673 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.219655 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" Oct 30 12:25:23.243586 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.243545 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst"] Oct 30 12:25:23.287947 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.287903 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh6jr\" (UniqueName: \"kubernetes.io/projected/edfbf552-0424-4e6c-bc8d-b0334b1cb737-kube-api-access-qh6jr\") pod \"maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst\" (UID: \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" Oct 30 12:25:23.288165 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.288046 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edfbf552-0424-4e6c-bc8d-b0334b1cb737-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst\" (UID: \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" Oct 30 12:25:23.288165 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.288122 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edfbf552-0424-4e6c-bc8d-b0334b1cb737-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst\" (UID: \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" Oct 30 12:25:23.288310 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.288280 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edfbf552-0424-4e6c-bc8d-b0334b1cb737-metrics-certs\") pod \"maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst\" (UID: \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" Oct 30 12:25:23.314178 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.314145 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8e3112-serving-cert\"" Oct 30 12:25:23.390192 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.390154 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh6jr\" (UniqueName: \"kubernetes.io/projected/edfbf552-0424-4e6c-bc8d-b0334b1cb737-kube-api-access-qh6jr\") pod \"maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst\" (UID: \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" Oct 30 12:25:23.390371 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.390240 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edfbf552-0424-4e6c-bc8d-b0334b1cb737-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst\" (UID: \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" Oct 30 12:25:23.390371 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.390270 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edfbf552-0424-4e6c-bc8d-b0334b1cb737-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst\" (UID: \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" Oct 30 12:25:23.390503 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.390406 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edfbf552-0424-4e6c-bc8d-b0334b1cb737-metrics-certs\") pod \"maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst\" (UID: \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" Oct 30 12:25:23.391015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.390995 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edfbf552-0424-4e6c-bc8d-b0334b1cb737-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst\" (UID: \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" Oct 30 12:25:23.392925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.392901 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edfbf552-0424-4e6c-bc8d-b0334b1cb737-metrics-certs\") pod \"maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst\" (UID: \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" Oct 30 12:25:23.393028 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.392911 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edfbf552-0424-4e6c-bc8d-b0334b1cb737-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst\" (UID: \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" Oct 30 12:25:23.414416 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.414391 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh6jr\" (UniqueName: \"kubernetes.io/projected/edfbf552-0424-4e6c-bc8d-b0334b1cb737-kube-api-access-qh6jr\") pod \"maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst\" (UID: \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" Oct 30 12:25:23.538542 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.538462 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" Oct 30 12:25:23.612145 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:25:23.612113 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:25:23.644799 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.644765 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" Oct 30 12:25:23.871146 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:25:23.871109 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedfbf552_0424_4e6c_bc8d_b0334b1cb737.slice/crio-36aec8d091a7a3a58982022188b0aa82cd239cab1e597aeb805388d9f0a757ee WatchSource:0}: Error finding container 36aec8d091a7a3a58982022188b0aa82cd239cab1e597aeb805388d9f0a757ee: Status 404 returned error can't find the container with id 36aec8d091a7a3a58982022188b0aa82cd239cab1e597aeb805388d9f0a757ee Oct 30 12:25:23.874176 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:23.874153 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst"] Oct 30 12:25:24.687743 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:24.687708 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" event={"ID":"edfbf552-0424-4e6c-bc8d-b0334b1cb737","Type":"ContainerStarted","Data":"2172f1a72a30b5c938404663b7bd45bfbb285bafeb42b1d7fc80b935e1202c1d"} Oct 30 12:25:24.688245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:24.687759 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" event={"ID":"edfbf552-0424-4e6c-bc8d-b0334b1cb737","Type":"ContainerStarted","Data":"36aec8d091a7a3a58982022188b0aa82cd239cab1e597aeb805388d9f0a757ee"} Oct 30 12:25:24.688245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:24.687962 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" Oct 30 12:25:24.741266 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:24.741208 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" podStartSLOduration=1.325269834 podStartE2EDuration="1.741191123s" totalImagesPullingTime="415.921289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:24.736626709 +0000 UTC m=+1875.770765106" watchObservedRunningTime="2026-10-30 12:25:24.741191123 +0000 UTC m=+1875.775329570" Oct 30 12:25:30.700800 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:30.700768 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" Oct 30 12:25:40.169240 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:40.169205 4468 generic.go:356] "Generic (PLEG): container finished" podID="51896ebb-81a0-4746-a5a1-22521ebdc85b" containerID="dc87487d2ae8cdf7ccad88fedc84bc2181ec547cd7897bc4a57b2c122dc93ec1" exitCode=0 Oct 30 12:25:40.169965 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:40.169277 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" event={"ID":"51896ebb-81a0-4746-a5a1-22521ebdc85b","Type":"ContainerDied","Data":"dc87487d2ae8cdf7ccad88fedc84bc2181ec547cd7897bc4a57b2c122dc93ec1"} Oct 30 12:25:40.169965 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:40.169801 4468 scope.go:122] "RemoveContainer" containerID="dc87487d2ae8cdf7ccad88fedc84bc2181ec547cd7897bc4a57b2c122dc93ec1" Oct 30 12:25:40.171548 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:40.171524 4468 generic.go:356] "Generic (PLEG): container finished" podID="efb7c9be-be70-4b37-8657-95c9a1e283ca" containerID="49918c97bcec47db9860abf33290cf90c23f9f1de00b1c6c8453578a4c4bbee0" exitCode=0 Oct 30 12:25:40.171677 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:40.171602 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" event={"ID":"efb7c9be-be70-4b37-8657-95c9a1e283ca","Type":"ContainerDied","Data":"49918c97bcec47db9860abf33290cf90c23f9f1de00b1c6c8453578a4c4bbee0"} Oct 30 12:25:40.172292 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:40.172269 4468 scope.go:122] "RemoveContainer" containerID="49918c97bcec47db9860abf33290cf90c23f9f1de00b1c6c8453578a4c4bbee0" Oct 30 12:25:40.172422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:40.172389 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:25:40.172494 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:40.172479 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:25:40.172535 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:40.172518 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:25:40.173919 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:40.173887 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:25:40.173991 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:40.173953 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:25:40.174032 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:40.173992 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:25:40.237344 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:40.231463 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:25:41.180785 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:41.180749 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" event={"ID":"51896ebb-81a0-4746-a5a1-22521ebdc85b","Type":"ContainerStarted","Data":"510ace7d7570fbbaf22c543c9649da10aa286520d709e5d332c6be0d6c505d35"} Oct 30 12:25:41.183344 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:41.183312 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" event={"ID":"efb7c9be-be70-4b37-8657-95c9a1e283ca","Type":"ContainerStarted","Data":"c599648ca47ec8982574e7c49174958e8d2cf23ca96ba7cc9cc6f51c8b3945ad"} Oct 30 12:25:41.184668 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:41.184649 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:25:41.186242 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:41.186223 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 12:25:41.186312 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:41.186266 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" podUID="efb7c9be-be70-4b37-8657-95c9a1e283ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 12:25:41.226940 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:41.224153 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:25:41.341091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:41.327479 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:25:41.354095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:41.348230 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 12:25:41.354095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:41.348301 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" podUID="51896ebb-81a0-4746-a5a1-22521ebdc85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 12:25:42.185113 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:42.185058 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 12:25:42.185542 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:42.185122 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" podUID="efb7c9be-be70-4b37-8657-95c9a1e283ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 12:25:42.327090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:42.325520 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 12:25:42.327090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:42.325589 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" podUID="51896ebb-81a0-4746-a5a1-22521ebdc85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 12:25:43.185307 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:43.185267 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 12:25:43.185884 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:43.185848 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" podUID="efb7c9be-be70-4b37-8657-95c9a1e283ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 12:25:43.349913 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:43.349872 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 12:25:43.350226 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:43.350194 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" podUID="51896ebb-81a0-4746-a5a1-22521ebdc85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 12:25:43.373279 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:43.371096 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6"] Oct 30 12:25:43.377223 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:43.377180 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx"] Oct 30 12:25:44.188922 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:44.188872 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 12:25:44.189584 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:44.189548 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" podUID="efb7c9be-be70-4b37-8657-95c9a1e283ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 12:25:44.329702 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:44.329667 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 12:25:44.330031 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:44.330004 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" podUID="51896ebb-81a0-4746-a5a1-22521ebdc85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 12:25:45.185325 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:45.185290 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 12:25:45.185511 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:45.185342 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" podUID="efb7c9be-be70-4b37-8657-95c9a1e283ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 12:25:45.347118 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:45.344744 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 12:25:45.347118 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:45.344812 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" podUID="51896ebb-81a0-4746-a5a1-22521ebdc85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 12:25:45.615222 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:45.614890 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz"] Oct 30 12:25:45.683154 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:45.675754 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn"] Oct 30 12:25:46.184742 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:46.184692 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 12:25:46.184963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:46.184755 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" podUID="efb7c9be-be70-4b37-8657-95c9a1e283ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 12:25:46.348095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:46.347081 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 12:25:46.348095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:46.347151 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" podUID="51896ebb-81a0-4746-a5a1-22521ebdc85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 12:25:47.185198 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:47.185168 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 12:25:47.185387 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:47.185212 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" podUID="efb7c9be-be70-4b37-8657-95c9a1e283ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 12:25:47.324535 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:47.324496 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 12:25:47.324747 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:47.324568 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" podUID="51896ebb-81a0-4746-a5a1-22521ebdc85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 12:25:48.189702 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.189669 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:25:48.190118 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.190087 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:25:48.190694 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.190672 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:25:48.331006 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.330973 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:25:48.334350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.334311 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:25:48.371857 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.371813 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6" podUID="4bfe187d-97e5-45cb-b643-4a8946b59df1" containerName="payload-processing" containerID="cri-o://dcb32b1ca11dd741a80c6666c6d8b2199746e7c2079f0e4572615ecee2067fef" gracePeriod=25 Oct 30 12:25:48.378525 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.378488 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx" podUID="3fc01cbd-3d3d-46db-81e9-67dc614b7205" containerName="payload-pre-processing" containerID="cri-o://2fb9b8e7e8e2dd3bcffb357ef333f590ddd908046c47f97730f9bcfb268f09e3" gracePeriod=25 Oct 30 12:25:48.851331 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.851294 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6" Oct 30 12:25:48.871147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.871115 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx" Oct 30 12:25:48.887851 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.887814 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fc01cbd-3d3d-46db-81e9-67dc614b7205-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fc01cbd-3d3d-46db-81e9-67dc614b7205-plugins-config-volume\") pod \"3fc01cbd-3d3d-46db-81e9-67dc614b7205\" (UID: \"3fc01cbd-3d3d-46db-81e9-67dc614b7205\") " Oct 30 12:25:48.888038 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.887959 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bfe187d-97e5-45cb-b643-4a8946b59df1-kube-api-access-s7zp6\" (UniqueName: \"kubernetes.io/projected/4bfe187d-97e5-45cb-b643-4a8946b59df1-kube-api-access-s7zp6\") pod \"4bfe187d-97e5-45cb-b643-4a8946b59df1\" (UID: \"4bfe187d-97e5-45cb-b643-4a8946b59df1\") " Oct 30 12:25:48.888038 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.888001 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc01cbd-3d3d-46db-81e9-67dc614b7205-kube-api-access-fhjcm\" (UniqueName: \"kubernetes.io/projected/3fc01cbd-3d3d-46db-81e9-67dc614b7205-kube-api-access-fhjcm\") pod \"3fc01cbd-3d3d-46db-81e9-67dc614b7205\" (UID: \"3fc01cbd-3d3d-46db-81e9-67dc614b7205\") " Oct 30 12:25:48.888038 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.888025 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bfe187d-97e5-45cb-b643-4a8946b59df1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bfe187d-97e5-45cb-b643-4a8946b59df1-plugins-config-volume\") pod \"4bfe187d-97e5-45cb-b643-4a8946b59df1\" (UID: \"4bfe187d-97e5-45cb-b643-4a8946b59df1\") " Oct 30 12:25:48.888520 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.888490 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc01cbd-3d3d-46db-81e9-67dc614b7205-plugins-config-volume" pod "3fc01cbd-3d3d-46db-81e9-67dc614b7205" (UID: "3fc01cbd-3d3d-46db-81e9-67dc614b7205"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:48.888838 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.888736 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bfe187d-97e5-45cb-b643-4a8946b59df1-plugins-config-volume" pod "4bfe187d-97e5-45cb-b643-4a8946b59df1" (UID: "4bfe187d-97e5-45cb-b643-4a8946b59df1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:48.890420 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.890397 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc01cbd-3d3d-46db-81e9-67dc614b7205-kube-api-access-fhjcm" pod "3fc01cbd-3d3d-46db-81e9-67dc614b7205" (UID: "3fc01cbd-3d3d-46db-81e9-67dc614b7205"). InnerVolumeSpecName "kube-api-access-fhjcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:48.890489 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.890418 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bfe187d-97e5-45cb-b643-4a8946b59df1-kube-api-access-s7zp6" pod "4bfe187d-97e5-45cb-b643-4a8946b59df1" (UID: "4bfe187d-97e5-45cb-b643-4a8946b59df1"). InnerVolumeSpecName "kube-api-access-s7zp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:48.990211 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.990174 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fc01cbd-3d3d-46db-81e9-67dc614b7205-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:48.990211 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.990212 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7zp6\" (UniqueName: \"kubernetes.io/projected/4bfe187d-97e5-45cb-b643-4a8946b59df1-kube-api-access-s7zp6\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:48.990211 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.990228 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhjcm\" (UniqueName: \"kubernetes.io/projected/3fc01cbd-3d3d-46db-81e9-67dc614b7205-kube-api-access-fhjcm\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:48.990550 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:48.990242 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bfe187d-97e5-45cb-b643-4a8946b59df1-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:49.300617 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:49.300580 4468 generic.go:356] "Generic (PLEG): container finished" podID="4bfe187d-97e5-45cb-b643-4a8946b59df1" containerID="dcb32b1ca11dd741a80c6666c6d8b2199746e7c2079f0e4572615ecee2067fef" exitCode=0 Oct 30 12:25:49.301113 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:49.300644 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6" Oct 30 12:25:49.301113 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:49.300668 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6" event={"ID":"4bfe187d-97e5-45cb-b643-4a8946b59df1","Type":"ContainerDied","Data":"dcb32b1ca11dd741a80c6666c6d8b2199746e7c2079f0e4572615ecee2067fef"} Oct 30 12:25:49.301113 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:49.300765 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6" event={"ID":"4bfe187d-97e5-45cb-b643-4a8946b59df1","Type":"ContainerDied","Data":"38689b86eb376b139d8a61443a087ba62a522262df4eed5f7c1c571d7d4c6283"} Oct 30 12:25:49.301113 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:49.300784 4468 scope.go:122] "RemoveContainer" containerID="dcb32b1ca11dd741a80c6666c6d8b2199746e7c2079f0e4572615ecee2067fef" Oct 30 12:25:49.302820 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:49.302784 4468 generic.go:356] "Generic (PLEG): container finished" podID="3fc01cbd-3d3d-46db-81e9-67dc614b7205" containerID="2fb9b8e7e8e2dd3bcffb357ef333f590ddd908046c47f97730f9bcfb268f09e3" exitCode=0 Oct 30 12:25:49.302971 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:49.302852 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx" Oct 30 12:25:49.302971 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:49.302934 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx" event={"ID":"3fc01cbd-3d3d-46db-81e9-67dc614b7205","Type":"ContainerDied","Data":"2fb9b8e7e8e2dd3bcffb357ef333f590ddd908046c47f97730f9bcfb268f09e3"} Oct 30 12:25:49.303131 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:49.302978 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx" event={"ID":"3fc01cbd-3d3d-46db-81e9-67dc614b7205","Type":"ContainerDied","Data":"74b70c9404ea44f14c0c71354855ad15460b34e34a31e605fb191a187c20492d"} Oct 30 12:25:49.314191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:49.314175 4468 scope.go:122] "RemoveContainer" containerID="dcb32b1ca11dd741a80c6666c6d8b2199746e7c2079f0e4572615ecee2067fef" Oct 30 12:25:49.314481 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:25:49.314463 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcb32b1ca11dd741a80c6666c6d8b2199746e7c2079f0e4572615ecee2067fef\": container with ID starting with dcb32b1ca11dd741a80c6666c6d8b2199746e7c2079f0e4572615ecee2067fef not found: ID does not exist" containerID="dcb32b1ca11dd741a80c6666c6d8b2199746e7c2079f0e4572615ecee2067fef" Oct 30 12:25:49.314539 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:49.314489 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcb32b1ca11dd741a80c6666c6d8b2199746e7c2079f0e4572615ecee2067fef"} err="failed to get container status \"dcb32b1ca11dd741a80c6666c6d8b2199746e7c2079f0e4572615ecee2067fef\": rpc error: code = NotFound desc = could not find container \"dcb32b1ca11dd741a80c6666c6d8b2199746e7c2079f0e4572615ecee2067fef\": container with ID starting with dcb32b1ca11dd741a80c6666c6d8b2199746e7c2079f0e4572615ecee2067fef not found: ID does not exist" Oct 30 12:25:49.314539 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:49.314506 4468 scope.go:122] "RemoveContainer" containerID="2fb9b8e7e8e2dd3bcffb357ef333f590ddd908046c47f97730f9bcfb268f09e3" Oct 30 12:25:49.323905 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:49.323889 4468 scope.go:122] "RemoveContainer" containerID="2fb9b8e7e8e2dd3bcffb357ef333f590ddd908046c47f97730f9bcfb268f09e3" Oct 30 12:25:49.324225 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:25:49.324201 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fb9b8e7e8e2dd3bcffb357ef333f590ddd908046c47f97730f9bcfb268f09e3\": container with ID starting with 2fb9b8e7e8e2dd3bcffb357ef333f590ddd908046c47f97730f9bcfb268f09e3 not found: ID does not exist" containerID="2fb9b8e7e8e2dd3bcffb357ef333f590ddd908046c47f97730f9bcfb268f09e3" Oct 30 12:25:49.324279 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:49.324234 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb9b8e7e8e2dd3bcffb357ef333f590ddd908046c47f97730f9bcfb268f09e3"} err="failed to get container status \"2fb9b8e7e8e2dd3bcffb357ef333f590ddd908046c47f97730f9bcfb268f09e3\": rpc error: code = NotFound desc = could not find container \"2fb9b8e7e8e2dd3bcffb357ef333f590ddd908046c47f97730f9bcfb268f09e3\": container with ID starting with 2fb9b8e7e8e2dd3bcffb357ef333f590ddd908046c47f97730f9bcfb268f09e3 not found: ID does not exist" Oct 30 12:25:49.360850 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:49.360759 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx"] Oct 30 12:25:49.372730 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:49.372692 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-69cc8bd9c5-zhnhx"] Oct 30 12:25:49.416164 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:49.416134 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6"] Oct 30 12:25:49.420684 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:49.420657 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-876549-7c67b6f94b-n2qg6"] Oct 30 12:25:50.675090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:50.667496 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz" podUID="6a13adda-0849-4688-a331-04b6701a8325" containerName="payload-processing" containerID="cri-o://1645fa0ca63621780d7458b320b5301d8e47febc0c9aaadf7f8462e4c58d233a" gracePeriod=25 Oct 30 12:25:50.685440 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:50.684354 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn" podUID="82ada6d9-8378-48b5-aac2-38294aa6ef5c" containerName="payload-pre-processing" containerID="cri-o://0aa1853dbf536b8cfad6a04605fa0f4b9e380dab3ee0cc16b008f09e9f642c07" gracePeriod=25 Oct 30 12:25:50.998200 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:50.998171 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz" Oct 30 12:25:51.019135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.019111 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn" Oct 30 12:25:51.081636 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.081595 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a13adda-0849-4688-a331-04b6701a8325-kube-api-access-z9sj5\" (UniqueName: \"kubernetes.io/projected/6a13adda-0849-4688-a331-04b6701a8325-kube-api-access-z9sj5\") pod \"6a13adda-0849-4688-a331-04b6701a8325\" (UID: \"6a13adda-0849-4688-a331-04b6701a8325\") " Oct 30 12:25:51.081834 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.081675 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ada6d9-8378-48b5-aac2-38294aa6ef5c-kube-api-access-27bdw\" (UniqueName: \"kubernetes.io/projected/82ada6d9-8378-48b5-aac2-38294aa6ef5c-kube-api-access-27bdw\") pod \"82ada6d9-8378-48b5-aac2-38294aa6ef5c\" (UID: \"82ada6d9-8378-48b5-aac2-38294aa6ef5c\") " Oct 30 12:25:51.081834 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.081826 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a13adda-0849-4688-a331-04b6701a8325-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a13adda-0849-4688-a331-04b6701a8325-plugins-config-volume\") pod \"6a13adda-0849-4688-a331-04b6701a8325\" (UID: \"6a13adda-0849-4688-a331-04b6701a8325\") " Oct 30 12:25:51.081985 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.081854 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ada6d9-8378-48b5-aac2-38294aa6ef5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ada6d9-8378-48b5-aac2-38294aa6ef5c-plugins-config-volume\") pod \"82ada6d9-8378-48b5-aac2-38294aa6ef5c\" (UID: \"82ada6d9-8378-48b5-aac2-38294aa6ef5c\") " Oct 30 12:25:51.082831 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.082797 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ada6d9-8378-48b5-aac2-38294aa6ef5c-plugins-config-volume" pod "82ada6d9-8378-48b5-aac2-38294aa6ef5c" (UID: "82ada6d9-8378-48b5-aac2-38294aa6ef5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:51.084025 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.083995 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a13adda-0849-4688-a331-04b6701a8325-plugins-config-volume" pod "6a13adda-0849-4688-a331-04b6701a8325" (UID: "6a13adda-0849-4688-a331-04b6701a8325"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:51.085362 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.085331 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ada6d9-8378-48b5-aac2-38294aa6ef5c-kube-api-access-27bdw" pod "82ada6d9-8378-48b5-aac2-38294aa6ef5c" (UID: "82ada6d9-8378-48b5-aac2-38294aa6ef5c"). InnerVolumeSpecName "kube-api-access-27bdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:51.085468 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.085404 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a13adda-0849-4688-a331-04b6701a8325-kube-api-access-z9sj5" pod "6a13adda-0849-4688-a331-04b6701a8325" (UID: "6a13adda-0849-4688-a331-04b6701a8325"). InnerVolumeSpecName "kube-api-access-z9sj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:51.099398 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.099323 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc01cbd-3d3d-46db-81e9-67dc614b7205" path="/var/lib/kubelet/pods/3fc01cbd-3d3d-46db-81e9-67dc614b7205/volumes" Oct 30 12:25:51.099974 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.099947 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bfe187d-97e5-45cb-b643-4a8946b59df1" path="/var/lib/kubelet/pods/4bfe187d-97e5-45cb-b643-4a8946b59df1/volumes" Oct 30 12:25:51.200094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.193657 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9sj5\" (UniqueName: \"kubernetes.io/projected/6a13adda-0849-4688-a331-04b6701a8325-kube-api-access-z9sj5\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:51.200094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.193701 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27bdw\" (UniqueName: \"kubernetes.io/projected/82ada6d9-8378-48b5-aac2-38294aa6ef5c-kube-api-access-27bdw\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:51.226148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.225144 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a13adda-0849-4688-a331-04b6701a8325-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:51.226148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.225197 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ada6d9-8378-48b5-aac2-38294aa6ef5c-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:51.519108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.519009 4468 generic.go:356] "Generic (PLEG): container finished" podID="6a13adda-0849-4688-a331-04b6701a8325" containerID="1645fa0ca63621780d7458b320b5301d8e47febc0c9aaadf7f8462e4c58d233a" exitCode=0 Oct 30 12:25:51.519108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.519054 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz" Oct 30 12:25:51.519341 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.519087 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz" event={"ID":"6a13adda-0849-4688-a331-04b6701a8325","Type":"ContainerDied","Data":"1645fa0ca63621780d7458b320b5301d8e47febc0c9aaadf7f8462e4c58d233a"} Oct 30 12:25:51.519341 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.519217 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz" event={"ID":"6a13adda-0849-4688-a331-04b6701a8325","Type":"ContainerDied","Data":"d412d89c1d67fcdc256adb1416639c5e9f83451f5ebceb9165b00f4ea3f011a0"} Oct 30 12:25:51.519341 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.519244 4468 scope.go:122] "RemoveContainer" containerID="1645fa0ca63621780d7458b320b5301d8e47febc0c9aaadf7f8462e4c58d233a" Oct 30 12:25:51.522359 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.522328 4468 generic.go:356] "Generic (PLEG): container finished" podID="82ada6d9-8378-48b5-aac2-38294aa6ef5c" containerID="0aa1853dbf536b8cfad6a04605fa0f4b9e380dab3ee0cc16b008f09e9f642c07" exitCode=0 Oct 30 12:25:51.522469 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.522373 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn" Oct 30 12:25:51.522469 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.522426 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn" event={"ID":"82ada6d9-8378-48b5-aac2-38294aa6ef5c","Type":"ContainerDied","Data":"0aa1853dbf536b8cfad6a04605fa0f4b9e380dab3ee0cc16b008f09e9f642c07"} Oct 30 12:25:51.522553 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.522489 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn" event={"ID":"82ada6d9-8378-48b5-aac2-38294aa6ef5c","Type":"ContainerDied","Data":"d68bcb98cbbf64af34f5bc7657d7e024183f9972431c8c6ea8e54d641b7b8e31"} Oct 30 12:25:51.531809 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.531781 4468 scope.go:122] "RemoveContainer" containerID="1645fa0ca63621780d7458b320b5301d8e47febc0c9aaadf7f8462e4c58d233a" Oct 30 12:25:51.532196 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:25:51.532141 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1645fa0ca63621780d7458b320b5301d8e47febc0c9aaadf7f8462e4c58d233a\": container with ID starting with 1645fa0ca63621780d7458b320b5301d8e47febc0c9aaadf7f8462e4c58d233a not found: ID does not exist" containerID="1645fa0ca63621780d7458b320b5301d8e47febc0c9aaadf7f8462e4c58d233a" Oct 30 12:25:51.532196 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.532175 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1645fa0ca63621780d7458b320b5301d8e47febc0c9aaadf7f8462e4c58d233a"} err="failed to get container status \"1645fa0ca63621780d7458b320b5301d8e47febc0c9aaadf7f8462e4c58d233a\": rpc error: code = NotFound desc = could not find container \"1645fa0ca63621780d7458b320b5301d8e47febc0c9aaadf7f8462e4c58d233a\": container with ID starting with 1645fa0ca63621780d7458b320b5301d8e47febc0c9aaadf7f8462e4c58d233a not found: ID does not exist" Oct 30 12:25:51.532196 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.532195 4468 scope.go:122] "RemoveContainer" containerID="0aa1853dbf536b8cfad6a04605fa0f4b9e380dab3ee0cc16b008f09e9f642c07" Oct 30 12:25:51.545492 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.545464 4468 scope.go:122] "RemoveContainer" containerID="0aa1853dbf536b8cfad6a04605fa0f4b9e380dab3ee0cc16b008f09e9f642c07" Oct 30 12:25:51.545925 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:25:51.545898 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa1853dbf536b8cfad6a04605fa0f4b9e380dab3ee0cc16b008f09e9f642c07\": container with ID starting with 0aa1853dbf536b8cfad6a04605fa0f4b9e380dab3ee0cc16b008f09e9f642c07 not found: ID does not exist" containerID="0aa1853dbf536b8cfad6a04605fa0f4b9e380dab3ee0cc16b008f09e9f642c07" Oct 30 12:25:51.546007 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.545940 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa1853dbf536b8cfad6a04605fa0f4b9e380dab3ee0cc16b008f09e9f642c07"} err="failed to get container status \"0aa1853dbf536b8cfad6a04605fa0f4b9e380dab3ee0cc16b008f09e9f642c07\": rpc error: code = NotFound desc = could not find container \"0aa1853dbf536b8cfad6a04605fa0f4b9e380dab3ee0cc16b008f09e9f642c07\": container with ID starting with 0aa1853dbf536b8cfad6a04605fa0f4b9e380dab3ee0cc16b008f09e9f642c07 not found: ID does not exist" Oct 30 12:25:51.586275 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.586239 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz"] Oct 30 12:25:51.601558 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.601527 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-6dff647484-mkkxz"] Oct 30 12:25:51.648240 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.648195 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn"] Oct 30 12:25:51.662518 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:51.662474 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-5d65fc55f-6vvfn"] Oct 30 12:25:53.091737 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:53.091704 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a13adda-0849-4688-a331-04b6701a8325" path="/var/lib/kubelet/pods/6a13adda-0849-4688-a331-04b6701a8325/volumes" Oct 30 12:25:53.092217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:53.092201 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82ada6d9-8378-48b5-aac2-38294aa6ef5c" path="/var/lib/kubelet/pods/82ada6d9-8378-48b5-aac2-38294aa6ef5c/volumes" Oct 30 12:25:53.600289 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:25:53.600254 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:25:54.077182 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:54.077143 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47"] Oct 30 12:25:54.211930 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:54.211896 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh"] Oct 30 12:25:58.031742 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.031700 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn"] Oct 30 12:25:58.033146 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.033115 4468 cpu_manager.go:414] "containerMap: removing container" podUID="4bfe187d-97e5-45cb-b643-4a8946b59df1" containerName="payload-processing" Oct 30 12:25:58.033146 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.033144 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bfe187d-97e5-45cb-b643-4a8946b59df1" containerName="payload-processing" Oct 30 12:25:58.033335 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.033179 4468 cpu_manager.go:414] "containerMap: removing container" podUID="82ada6d9-8378-48b5-aac2-38294aa6ef5c" containerName="payload-pre-processing" Oct 30 12:25:58.033335 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.033186 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ada6d9-8378-48b5-aac2-38294aa6ef5c" containerName="payload-pre-processing" Oct 30 12:25:58.033335 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.033215 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3fc01cbd-3d3d-46db-81e9-67dc614b7205" containerName="payload-pre-processing" Oct 30 12:25:58.033335 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.033223 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc01cbd-3d3d-46db-81e9-67dc614b7205" containerName="payload-pre-processing" Oct 30 12:25:58.033335 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.033245 4468 cpu_manager.go:414] "containerMap: removing container" podUID="6a13adda-0849-4688-a331-04b6701a8325" containerName="payload-processing" Oct 30 12:25:58.033335 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.033252 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a13adda-0849-4688-a331-04b6701a8325" containerName="payload-processing" Oct 30 12:25:58.033633 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.033459 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc01cbd-3d3d-46db-81e9-67dc614b7205" containerName="payload-pre-processing" Oct 30 12:25:58.033633 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.033479 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a13adda-0849-4688-a331-04b6701a8325" containerName="payload-processing" Oct 30 12:25:58.033633 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.033496 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bfe187d-97e5-45cb-b643-4a8946b59df1" containerName="payload-processing" Oct 30 12:25:58.033633 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.033510 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ada6d9-8378-48b5-aac2-38294aa6ef5c" containerName="payload-pre-processing" Oct 30 12:25:58.039194 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.039016 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn" Oct 30 12:25:58.047540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.047508 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn"] Oct 30 12:25:58.063010 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.059317 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-266486\"" Oct 30 12:25:58.135517 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.129423 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-266486-dockercfg-ch5bx\"" Oct 30 12:25:58.386489 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.386389 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xfzt\" (UniqueName: \"kubernetes.io/projected/58c942ca-8c8b-4e96-b2ed-e2cbe45af71c-kube-api-access-9xfzt\") pod \"payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn\" (UID: \"58c942ca-8c8b-4e96-b2ed-e2cbe45af71c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn" Oct 30 12:25:58.386860 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.386826 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c942ca-8c8b-4e96-b2ed-e2cbe45af71c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn\" (UID: \"58c942ca-8c8b-4e96-b2ed-e2cbe45af71c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn" Oct 30 12:25:58.488993 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.488944 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c942ca-8c8b-4e96-b2ed-e2cbe45af71c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn\" (UID: \"58c942ca-8c8b-4e96-b2ed-e2cbe45af71c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn" Oct 30 12:25:58.489220 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.489142 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xfzt\" (UniqueName: \"kubernetes.io/projected/58c942ca-8c8b-4e96-b2ed-e2cbe45af71c-kube-api-access-9xfzt\") pod \"payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn\" (UID: \"58c942ca-8c8b-4e96-b2ed-e2cbe45af71c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn" Oct 30 12:25:58.489723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.489701 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c942ca-8c8b-4e96-b2ed-e2cbe45af71c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn\" (UID: \"58c942ca-8c8b-4e96-b2ed-e2cbe45af71c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn" Oct 30 12:25:58.531093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.518930 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xfzt\" (UniqueName: \"kubernetes.io/projected/58c942ca-8c8b-4e96-b2ed-e2cbe45af71c-kube-api-access-9xfzt\") pod \"payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn\" (UID: \"58c942ca-8c8b-4e96-b2ed-e2cbe45af71c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn" Oct 30 12:25:58.584240 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.584207 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn" Oct 30 12:25:58.933219 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:58.933185 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn"] Oct 30 12:25:58.935514 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:25:58.935489 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c942ca_8c8b_4e96_b2ed_e2cbe45af71c.slice/crio-52f7a718a9bfcfd705857fac3283ecbe9f662c3e8dbd09f6e92120bb89687ac1 WatchSource:0}: Error finding container 52f7a718a9bfcfd705857fac3283ecbe9f662c3e8dbd09f6e92120bb89687ac1: Status 404 returned error can't find the container with id 52f7a718a9bfcfd705857fac3283ecbe9f662c3e8dbd09f6e92120bb89687ac1 Oct 30 12:25:59.077917 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.077871 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47" podUID="9868d552-736b-4752-9272-8805b53a8dcf" containerName="payload-processing" containerID="cri-o://05b1a9734ac39896e81c728cd91aac4f86c00b0e727d1e9b3b1bb7bffbcfc935" gracePeriod=25 Oct 30 12:25:59.219093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.217147 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh" podUID="60785db9-4e26-4614-ab7c-490407d0095b" containerName="payload-pre-processing" containerID="cri-o://5ddded07342ba731829dac140c82c1e15bd4fd671f54b632bb14de53de00e186" gracePeriod=25 Oct 30 12:25:59.385994 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.385966 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47" Oct 30 12:25:59.402943 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.402913 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9868d552-736b-4752-9272-8805b53a8dcf-kube-api-access-7pcs6\" (UniqueName: \"kubernetes.io/projected/9868d552-736b-4752-9272-8805b53a8dcf-kube-api-access-7pcs6\") pod \"9868d552-736b-4752-9272-8805b53a8dcf\" (UID: \"9868d552-736b-4752-9272-8805b53a8dcf\") " Oct 30 12:25:59.403119 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.402997 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9868d552-736b-4752-9272-8805b53a8dcf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9868d552-736b-4752-9272-8805b53a8dcf-plugins-config-volume\") pod \"9868d552-736b-4752-9272-8805b53a8dcf\" (UID: \"9868d552-736b-4752-9272-8805b53a8dcf\") " Oct 30 12:25:59.425367 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.425304 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9868d552-736b-4752-9272-8805b53a8dcf-plugins-config-volume" pod "9868d552-736b-4752-9272-8805b53a8dcf" (UID: "9868d552-736b-4752-9272-8805b53a8dcf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:59.437264 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.437207 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9868d552-736b-4752-9272-8805b53a8dcf-kube-api-access-7pcs6" pod "9868d552-736b-4752-9272-8805b53a8dcf" (UID: "9868d552-736b-4752-9272-8805b53a8dcf"). InnerVolumeSpecName "kube-api-access-7pcs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:59.527154 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.526670 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9868d552-736b-4752-9272-8805b53a8dcf-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:59.527154 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.526711 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pcs6\" (UniqueName: \"kubernetes.io/projected/9868d552-736b-4752-9272-8805b53a8dcf-kube-api-access-7pcs6\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:59.581138 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.581111 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh" Oct 30 12:25:59.610095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.603088 4468 generic.go:356] "Generic (PLEG): container finished" podID="9868d552-736b-4752-9272-8805b53a8dcf" containerID="05b1a9734ac39896e81c728cd91aac4f86c00b0e727d1e9b3b1bb7bffbcfc935" exitCode=0 Oct 30 12:25:59.610095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.603132 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47" event={"ID":"9868d552-736b-4752-9272-8805b53a8dcf","Type":"ContainerDied","Data":"05b1a9734ac39896e81c728cd91aac4f86c00b0e727d1e9b3b1bb7bffbcfc935"} Oct 30 12:25:59.610095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.603193 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47" Oct 30 12:25:59.610095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.603207 4468 scope.go:122] "RemoveContainer" containerID="05b1a9734ac39896e81c728cd91aac4f86c00b0e727d1e9b3b1bb7bffbcfc935" Oct 30 12:25:59.610460 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.603194 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47" event={"ID":"9868d552-736b-4752-9272-8805b53a8dcf","Type":"ContainerDied","Data":"3a248e032dac8ebc78b415ee0e2d09b2d13a7ce678588a16abb74c7ff02b72df"} Oct 30 12:25:59.632093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.628468 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60785db9-4e26-4614-ab7c-490407d0095b-kube-api-access-ztcwk\" (UniqueName: \"kubernetes.io/projected/60785db9-4e26-4614-ab7c-490407d0095b-kube-api-access-ztcwk\") pod \"60785db9-4e26-4614-ab7c-490407d0095b\" (UID: \"60785db9-4e26-4614-ab7c-490407d0095b\") " Oct 30 12:25:59.632093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.628543 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60785db9-4e26-4614-ab7c-490407d0095b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60785db9-4e26-4614-ab7c-490407d0095b-plugins-config-volume\") pod \"60785db9-4e26-4614-ab7c-490407d0095b\" (UID: \"60785db9-4e26-4614-ab7c-490407d0095b\") " Oct 30 12:25:59.635314 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.635277 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60785db9-4e26-4614-ab7c-490407d0095b-plugins-config-volume" pod "60785db9-4e26-4614-ab7c-490407d0095b" (UID: "60785db9-4e26-4614-ab7c-490407d0095b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:59.637660 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.637635 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60785db9-4e26-4614-ab7c-490407d0095b-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:59.640764 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.640576 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn" event={"ID":"58c942ca-8c8b-4e96-b2ed-e2cbe45af71c","Type":"ContainerStarted","Data":"aa7835a289d03145b929ef0bc6c231da5383806d8e4d56f60a2bde2969f24f69"} Oct 30 12:25:59.640764 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.640616 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60785db9-4e26-4614-ab7c-490407d0095b-kube-api-access-ztcwk" pod "60785db9-4e26-4614-ab7c-490407d0095b" (UID: "60785db9-4e26-4614-ab7c-490407d0095b"). InnerVolumeSpecName "kube-api-access-ztcwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:59.640764 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.640634 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn" event={"ID":"58c942ca-8c8b-4e96-b2ed-e2cbe45af71c","Type":"ContainerStarted","Data":"52f7a718a9bfcfd705857fac3283ecbe9f662c3e8dbd09f6e92120bb89687ac1"} Oct 30 12:25:59.641007 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.640921 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn" Oct 30 12:25:59.642188 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.642161 4468 generic.go:356] "Generic (PLEG): container finished" podID="60785db9-4e26-4614-ab7c-490407d0095b" containerID="5ddded07342ba731829dac140c82c1e15bd4fd671f54b632bb14de53de00e186" exitCode=0 Oct 30 12:25:59.642274 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.642220 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh" event={"ID":"60785db9-4e26-4614-ab7c-490407d0095b","Type":"ContainerDied","Data":"5ddded07342ba731829dac140c82c1e15bd4fd671f54b632bb14de53de00e186"} Oct 30 12:25:59.642274 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.642266 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh" event={"ID":"60785db9-4e26-4614-ab7c-490407d0095b","Type":"ContainerDied","Data":"dbf4315dddd7781bd855d48aeef9c4306e82950789a6780f86b8be58dae9cb5e"} Oct 30 12:25:59.642388 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.642293 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh" Oct 30 12:25:59.680778 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.680748 4468 scope.go:122] "RemoveContainer" containerID="05b1a9734ac39896e81c728cd91aac4f86c00b0e727d1e9b3b1bb7bffbcfc935" Oct 30 12:25:59.681409 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:25:59.681375 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b1a9734ac39896e81c728cd91aac4f86c00b0e727d1e9b3b1bb7bffbcfc935\": container with ID starting with 05b1a9734ac39896e81c728cd91aac4f86c00b0e727d1e9b3b1bb7bffbcfc935 not found: ID does not exist" containerID="05b1a9734ac39896e81c728cd91aac4f86c00b0e727d1e9b3b1bb7bffbcfc935" Oct 30 12:25:59.681532 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.681422 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b1a9734ac39896e81c728cd91aac4f86c00b0e727d1e9b3b1bb7bffbcfc935"} err="failed to get container status \"05b1a9734ac39896e81c728cd91aac4f86c00b0e727d1e9b3b1bb7bffbcfc935\": rpc error: code = NotFound desc = could not find container \"05b1a9734ac39896e81c728cd91aac4f86c00b0e727d1e9b3b1bb7bffbcfc935\": container with ID starting with 05b1a9734ac39896e81c728cd91aac4f86c00b0e727d1e9b3b1bb7bffbcfc935 not found: ID does not exist" Oct 30 12:25:59.681532 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.681458 4468 scope.go:122] "RemoveContainer" containerID="5ddded07342ba731829dac140c82c1e15bd4fd671f54b632bb14de53de00e186" Oct 30 12:25:59.696522 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.696501 4468 scope.go:122] "RemoveContainer" containerID="5ddded07342ba731829dac140c82c1e15bd4fd671f54b632bb14de53de00e186" Oct 30 12:25:59.696804 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:25:59.696781 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ddded07342ba731829dac140c82c1e15bd4fd671f54b632bb14de53de00e186\": container with ID starting with 5ddded07342ba731829dac140c82c1e15bd4fd671f54b632bb14de53de00e186 not found: ID does not exist" containerID="5ddded07342ba731829dac140c82c1e15bd4fd671f54b632bb14de53de00e186" Oct 30 12:25:59.696909 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.696814 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ddded07342ba731829dac140c82c1e15bd4fd671f54b632bb14de53de00e186"} err="failed to get container status \"5ddded07342ba731829dac140c82c1e15bd4fd671f54b632bb14de53de00e186\": rpc error: code = NotFound desc = could not find container \"5ddded07342ba731829dac140c82c1e15bd4fd671f54b632bb14de53de00e186\": container with ID starting with 5ddded07342ba731829dac140c82c1e15bd4fd671f54b632bb14de53de00e186 not found: ID does not exist" Oct 30 12:25:59.740390 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.740334 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn" podStartSLOduration=2.740317315 podStartE2EDuration="2.740317315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:59.736463437 +0000 UTC m=+1910.770601834" watchObservedRunningTime="2026-10-30 12:25:59.740317315 +0000 UTC m=+1910.774455745" Oct 30 12:25:59.744525 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.744482 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztcwk\" (UniqueName: \"kubernetes.io/projected/60785db9-4e26-4614-ab7c-490407d0095b-kube-api-access-ztcwk\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:25:59.779905 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.779821 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47"] Oct 30 12:25:59.784410 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.784385 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3112-f8589c488-qnl47"] Oct 30 12:25:59.868096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.852259 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh"] Oct 30 12:25:59.868096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:25:59.864604 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-7b956b5bf5-q56rh"] Oct 30 12:26:01.091022 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:01.090989 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60785db9-4e26-4614-ab7c-490407d0095b" path="/var/lib/kubelet/pods/60785db9-4e26-4614-ab7c-490407d0095b/volumes" Oct 30 12:26:01.091475 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:01.091459 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9868d552-736b-4752-9272-8805b53a8dcf" path="/var/lib/kubelet/pods/9868d552-736b-4752-9272-8805b53a8dcf/volumes" Oct 30 12:26:02.437296 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:02.437250 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:26:02.437716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:02.437347 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" podUID="df475a56-3557-40c3-84bb-0e812127ae21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:26:03.530100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.524609 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47"] Oct 30 12:26:03.530100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.525740 4468 cpu_manager.go:414] "containerMap: removing container" podUID="9868d552-736b-4752-9272-8805b53a8dcf" containerName="payload-processing" Oct 30 12:26:03.530100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.525765 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9868d552-736b-4752-9272-8805b53a8dcf" containerName="payload-processing" Oct 30 12:26:03.530100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.525813 4468 cpu_manager.go:414] "containerMap: removing container" podUID="60785db9-4e26-4614-ab7c-490407d0095b" containerName="payload-pre-processing" Oct 30 12:26:03.530100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.525822 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60785db9-4e26-4614-ab7c-490407d0095b" containerName="payload-pre-processing" Oct 30 12:26:03.530100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.526004 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="9868d552-736b-4752-9272-8805b53a8dcf" containerName="payload-processing" Oct 30 12:26:03.530100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.526021 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="60785db9-4e26-4614-ab7c-490407d0095b" containerName="payload-pre-processing" Oct 30 12:26:03.563092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.558480 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47"] Oct 30 12:26:03.563092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.558616 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47" Oct 30 12:26:03.605378 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.605338 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-876549\"" Oct 30 12:26:03.605579 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.605560 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-876549-dockercfg-kj4s2\"" Oct 30 12:26:03.703493 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.703334 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7243003a-ec74-47cd-9573-8083f2aeffd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47\" (UID: \"7243003a-ec74-47cd-9573-8083f2aeffd3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47" Oct 30 12:26:03.703822 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.703800 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7krnr\" (UniqueName: \"kubernetes.io/projected/7243003a-ec74-47cd-9573-8083f2aeffd3-kube-api-access-7krnr\") pod \"payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47\" (UID: \"7243003a-ec74-47cd-9573-8083f2aeffd3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47" Oct 30 12:26:03.728035 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.727995 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx"] Oct 30 12:26:03.731503 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.731482 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx" Oct 30 12:26:03.736217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.736183 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx"] Oct 30 12:26:03.805898 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.805811 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7krnr\" (UniqueName: \"kubernetes.io/projected/7243003a-ec74-47cd-9573-8083f2aeffd3-kube-api-access-7krnr\") pod \"payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47\" (UID: \"7243003a-ec74-47cd-9573-8083f2aeffd3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47" Oct 30 12:26:03.806085 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.806055 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7243003a-ec74-47cd-9573-8083f2aeffd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47\" (UID: \"7243003a-ec74-47cd-9573-8083f2aeffd3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47" Oct 30 12:26:03.806773 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.806754 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7243003a-ec74-47cd-9573-8083f2aeffd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47\" (UID: \"7243003a-ec74-47cd-9573-8083f2aeffd3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47" Oct 30 12:26:03.839621 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.839585 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7krnr\" (UniqueName: \"kubernetes.io/projected/7243003a-ec74-47cd-9573-8083f2aeffd3-kube-api-access-7krnr\") pod \"payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47\" (UID: \"7243003a-ec74-47cd-9573-8083f2aeffd3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47" Oct 30 12:26:03.866607 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.866553 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs"] Oct 30 12:26:03.867018 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.866815 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" podUID="53efa6ea-7b24-4df7-b9b3-46ab86493fc1" containerName="istio-proxy" containerID="cri-o://675403ae1d4426df2a949bddfa0468e39be553afe17710d97c22057e5ba0b1f8" gracePeriod=30 Oct 30 12:26:03.877328 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.877268 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml"] Oct 30 12:26:03.877676 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.877636 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" podUID="1d5f9e36-e943-405f-9f53-6ffb08d31c67" containerName="istio-proxy" containerID="cri-o://5bd0df2ad7d91628adc626180a351c50c0105e65f55437af0e794590526f2c20" gracePeriod=30 Oct 30 12:26:03.910295 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.907958 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptfwl\" (UniqueName: \"kubernetes.io/projected/fac8a1fa-9988-466f-ad04-5a09ff02763a-kube-api-access-ptfwl\") pod \"payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx\" (UID: \"fac8a1fa-9988-466f-ad04-5a09ff02763a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx" Oct 30 12:26:03.910295 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.908155 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac8a1fa-9988-466f-ad04-5a09ff02763a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx\" (UID: \"fac8a1fa-9988-466f-ad04-5a09ff02763a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx" Oct 30 12:26:03.994095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:03.992552 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47" Oct 30 12:26:04.010123 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:04.010085 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptfwl\" (UniqueName: \"kubernetes.io/projected/fac8a1fa-9988-466f-ad04-5a09ff02763a-kube-api-access-ptfwl\") pod \"payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx\" (UID: \"fac8a1fa-9988-466f-ad04-5a09ff02763a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx" Oct 30 12:26:04.010236 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:04.010223 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac8a1fa-9988-466f-ad04-5a09ff02763a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx\" (UID: \"fac8a1fa-9988-466f-ad04-5a09ff02763a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx" Oct 30 12:26:04.011198 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:04.011173 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac8a1fa-9988-466f-ad04-5a09ff02763a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx\" (UID: \"fac8a1fa-9988-466f-ad04-5a09ff02763a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx" Oct 30 12:26:04.038624 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:04.038525 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptfwl\" (UniqueName: \"kubernetes.io/projected/fac8a1fa-9988-466f-ad04-5a09ff02763a-kube-api-access-ptfwl\") pod \"payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx\" (UID: \"fac8a1fa-9988-466f-ad04-5a09ff02763a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx" Oct 30 12:26:04.049486 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:04.046460 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx" Oct 30 12:26:04.319900 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:26:04.319683 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7243003a_ec74_47cd_9573_8083f2aeffd3.slice/crio-943957dfa83a35494a083881e5037e8cb6d9f327baabd1dfa7a5c5d22d8f5e54 WatchSource:0}: Error finding container 943957dfa83a35494a083881e5037e8cb6d9f327baabd1dfa7a5c5d22d8f5e54: Status 404 returned error can't find the container with id 943957dfa83a35494a083881e5037e8cb6d9f327baabd1dfa7a5c5d22d8f5e54 Oct 30 12:26:04.323811 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:04.323785 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47"] Oct 30 12:26:04.400420 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:26:04.400389 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac8a1fa_9988_466f_ad04_5a09ff02763a.slice/crio-695fd13496760d433ed1b8380f9605ad757dcf5d31172640eb3ab55f47298a76 WatchSource:0}: Error finding container 695fd13496760d433ed1b8380f9605ad757dcf5d31172640eb3ab55f47298a76: Status 404 returned error can't find the container with id 695fd13496760d433ed1b8380f9605ad757dcf5d31172640eb3ab55f47298a76 Oct 30 12:26:04.403024 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:04.402996 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx"] Oct 30 12:26:04.736638 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:04.736601 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47" event={"ID":"7243003a-ec74-47cd-9573-8083f2aeffd3","Type":"ContainerStarted","Data":"943957dfa83a35494a083881e5037e8cb6d9f327baabd1dfa7a5c5d22d8f5e54"} Oct 30 12:26:04.738059 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:04.738034 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx" event={"ID":"fac8a1fa-9988-466f-ad04-5a09ff02763a","Type":"ContainerStarted","Data":"695fd13496760d433ed1b8380f9605ad757dcf5d31172640eb3ab55f47298a76"} Oct 30 12:26:05.751525 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:05.751423 4468 generic.go:356] "Generic (PLEG): container finished" podID="df475a56-3557-40c3-84bb-0e812127ae21" containerID="00f9b05fec387882161ab7e5a4de28ffd969d08326e5a0c690468e03a4d7a65a" exitCode=0 Oct 30 12:26:05.752155 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:05.751636 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" event={"ID":"df475a56-3557-40c3-84bb-0e812127ae21","Type":"ContainerDied","Data":"00f9b05fec387882161ab7e5a4de28ffd969d08326e5a0c690468e03a4d7a65a"} Oct 30 12:26:05.752235 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:05.752205 4468 scope.go:122] "RemoveContainer" containerID="00f9b05fec387882161ab7e5a4de28ffd969d08326e5a0c690468e03a4d7a65a" Oct 30 12:26:05.754613 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:05.754581 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:26:05.754720 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:05.754663 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:26:05.754720 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:05.754705 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:26:06.305785 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:06.301929 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6"] Oct 30 12:26:06.315352 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:06.313956 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6" Oct 30 12:26:06.334228 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:06.331024 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7e2db2-dockercfg-sbxh9\"" Oct 30 12:26:06.334228 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:06.332036 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6"] Oct 30 12:26:06.334228 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:06.332171 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7e2db2\"" Oct 30 12:26:06.467255 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:06.467214 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d14547-06cf-4583-b119-c8fbd7aa1e95-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6\" (UID: \"b8d14547-06cf-4583-b119-c8fbd7aa1e95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6" Oct 30 12:26:06.467456 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:06.467426 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p2lj\" (UniqueName: \"kubernetes.io/projected/b8d14547-06cf-4583-b119-c8fbd7aa1e95-kube-api-access-7p2lj\") pod \"payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6\" (UID: \"b8d14547-06cf-4583-b119-c8fbd7aa1e95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6" Oct 30 12:26:06.571133 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:06.569597 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d14547-06cf-4583-b119-c8fbd7aa1e95-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6\" (UID: \"b8d14547-06cf-4583-b119-c8fbd7aa1e95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6" Oct 30 12:26:06.571133 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:06.569812 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p2lj\" (UniqueName: \"kubernetes.io/projected/b8d14547-06cf-4583-b119-c8fbd7aa1e95-kube-api-access-7p2lj\") pod \"payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6\" (UID: \"b8d14547-06cf-4583-b119-c8fbd7aa1e95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6" Oct 30 12:26:06.571133 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:06.570593 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d14547-06cf-4583-b119-c8fbd7aa1e95-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6\" (UID: \"b8d14547-06cf-4583-b119-c8fbd7aa1e95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6" Oct 30 12:26:06.598585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:06.598548 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p2lj\" (UniqueName: \"kubernetes.io/projected/b8d14547-06cf-4583-b119-c8fbd7aa1e95-kube-api-access-7p2lj\") pod \"payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6\" (UID: \"b8d14547-06cf-4583-b119-c8fbd7aa1e95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6" Oct 30 12:26:06.639727 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:06.639689 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6" Oct 30 12:26:06.779627 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:06.779590 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" event={"ID":"df475a56-3557-40c3-84bb-0e812127ae21","Type":"ContainerStarted","Data":"173bebb153208b2cc8c4908438b281747f4cee56d3f5fac2c677ff7a30b0e1ec"} Oct 30 12:26:06.899802 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:06.899660 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:26:06.963886 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:06.963845 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6"] Oct 30 12:26:07.792907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:07.792799 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47" event={"ID":"7243003a-ec74-47cd-9573-8083f2aeffd3","Type":"ContainerStarted","Data":"750dc96ce6c6ad0afa4a09ea1610290ac318c44a48645c12e9656752d8b7bd69"} Oct 30 12:26:07.794743 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:07.794698 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47" Oct 30 12:26:07.815093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:07.802497 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx" event={"ID":"fac8a1fa-9988-466f-ad04-5a09ff02763a","Type":"ContainerStarted","Data":"84a47e70938500cf9fae500cdf137f8ec314acfa5b3584b0cc1296249ce9787b"} Oct 30 12:26:07.815093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:07.803365 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx" Oct 30 12:26:07.852651 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:07.852576 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47" podStartSLOduration=2.035487006 podStartE2EDuration="4.852558665s" totalImagesPullingTime="2.817071659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:07.851846913 +0000 UTC m=+1918.885985334" watchObservedRunningTime="2026-10-30 12:26:07.852558665 +0000 UTC m=+1918.886697063" Oct 30 12:26:07.863329 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:07.863284 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6" event={"ID":"b8d14547-06cf-4583-b119-c8fbd7aa1e95","Type":"ContainerStarted","Data":"182a3c8e6e917ae071b136110031a7e0201c928b6ad902a46eecba71d579652c"} Oct 30 12:26:07.864017 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:07.863997 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6" Oct 30 12:26:07.864670 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:07.864648 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6" event={"ID":"b8d14547-06cf-4583-b119-c8fbd7aa1e95","Type":"ContainerStarted","Data":"1d9f6e5a01016b7930461bbd136ec571bf931fc6483aa4a6dc907303f2edecd5"} Oct 30 12:26:07.899391 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:07.899324 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx" podStartSLOduration=2.139814407 podStartE2EDuration="4.899304518s" totalImagesPullingTime="2.759490111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:07.898373192 +0000 UTC m=+1918.932511587" watchObservedRunningTime="2026-10-30 12:26:07.899304518 +0000 UTC m=+1918.933442914" Oct 30 12:26:07.899596 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:07.899552 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 12:26:07.899671 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:07.899595 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" podUID="df475a56-3557-40c3-84bb-0e812127ae21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 12:26:07.960944 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:07.960884 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6" podStartSLOduration=1.496527981 podStartE2EDuration="1.960860097s" totalImagesPullingTime="464.332116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:07.953819561 +0000 UTC m=+1918.987957960" watchObservedRunningTime="2026-10-30 12:26:07.960860097 +0000 UTC m=+1918.994998523" Oct 30 12:26:08.901914 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:08.901883 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 12:26:08.902428 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:08.902397 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" podUID="df475a56-3557-40c3-84bb-0e812127ae21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 12:26:09.295490 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.295460 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:26:09.460145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.460112 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-envoy\") pod \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " Oct 30 12:26:09.460309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.460168 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-credential-socket\") pod \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " Oct 30 12:26:09.460309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.460207 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-workload-certs\") pod \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " Oct 30 12:26:09.460309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.460243 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d5f9e36-e943-405f-9f53-6ffb08d31c67-kube-api-access-pwk7n\" (UniqueName: \"kubernetes.io/projected/1d5f9e36-e943-405f-9f53-6ffb08d31c67-kube-api-access-pwk7n\") pod \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " Oct 30 12:26:09.460309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.460261 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-token\" (UniqueName: \"kubernetes.io/projected/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-token\") pod \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " Oct 30 12:26:09.460309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.460303 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-podinfo\") pod \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " Oct 30 12:26:09.460503 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.460367 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-data\") pod \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " Oct 30 12:26:09.460548 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.460541 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istiod-ca-cert\") pod \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " Oct 30 12:26:09.460601 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.460589 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-workload-socket\") pod \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\" (UID: \"1d5f9e36-e943-405f-9f53-6ffb08d31c67\") " Oct 30 12:26:09.461579 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.461543 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-credential-socket" pod "1d5f9e36-e943-405f-9f53-6ffb08d31c67" (UID: "1d5f9e36-e943-405f-9f53-6ffb08d31c67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:09.461743 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.461712 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-workload-socket" pod "1d5f9e36-e943-405f-9f53-6ffb08d31c67" (UID: "1d5f9e36-e943-405f-9f53-6ffb08d31c67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:09.461943 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.461922 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-data" pod "1d5f9e36-e943-405f-9f53-6ffb08d31c67" (UID: "1d5f9e36-e943-405f-9f53-6ffb08d31c67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:09.462242 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.462216 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istiod-ca-cert" pod "1d5f9e36-e943-405f-9f53-6ffb08d31c67" (UID: "1d5f9e36-e943-405f-9f53-6ffb08d31c67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:26:09.462493 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.462464 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-workload-certs" pod "1d5f9e36-e943-405f-9f53-6ffb08d31c67" (UID: "1d5f9e36-e943-405f-9f53-6ffb08d31c67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:09.469253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.468158 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-podinfo" pod "1d5f9e36-e943-405f-9f53-6ffb08d31c67" (UID: "1d5f9e36-e943-405f-9f53-6ffb08d31c67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:26:09.469253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.468297 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5f9e36-e943-405f-9f53-6ffb08d31c67-kube-api-access-pwk7n" pod "1d5f9e36-e943-405f-9f53-6ffb08d31c67" (UID: "1d5f9e36-e943-405f-9f53-6ffb08d31c67"). InnerVolumeSpecName "kube-api-access-pwk7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:26:09.469253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.468688 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-envoy" pod "1d5f9e36-e943-405f-9f53-6ffb08d31c67" (UID: "1d5f9e36-e943-405f-9f53-6ffb08d31c67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:09.470134 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.470101 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-token" pod "1d5f9e36-e943-405f-9f53-6ffb08d31c67" (UID: "1d5f9e36-e943-405f-9f53-6ffb08d31c67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:26:09.565975 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.565635 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.565975 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.565674 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.565975 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.565689 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.565975 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.565701 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.565975 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.565715 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.565975 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.565729 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d5f9e36-e943-405f-9f53-6ffb08d31c67-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.565975 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.565742 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwk7n\" (UniqueName: \"kubernetes.io/projected/1d5f9e36-e943-405f-9f53-6ffb08d31c67-kube-api-access-pwk7n\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.565975 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.565755 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.569121 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.568302 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d5f9e36-e943-405f-9f53-6ffb08d31c67-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.641475 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.641440 4468 scope.go:122] "RemoveContainer" containerID="5bd0df2ad7d91628adc626180a351c50c0105e65f55437af0e794590526f2c20" Oct 30 12:26:09.655837 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.655660 4468 scope.go:122] "RemoveContainer" containerID="87712a02ccb963f53d4cd38735ddea858f283e1c2fae805cd4a0237079f66ccb" Oct 30 12:26:09.660917 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.660889 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:26:09.770385 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.770349 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-workload-socket\") pod \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " Oct 30 12:26:09.770560 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.770447 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-envoy\") pod \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " Oct 30 12:26:09.770560 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.770499 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-data\") pod \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " Oct 30 12:26:09.770690 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.770565 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-workload-certs\") pod \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " Oct 30 12:26:09.770753 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.770678 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-workload-socket" pod "53efa6ea-7b24-4df7-b9b3-46ab86493fc1" (UID: "53efa6ea-7b24-4df7-b9b3-46ab86493fc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:09.770814 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.770790 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-token\" (UniqueName: \"kubernetes.io/projected/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-token\") pod \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " Oct 30 12:26:09.770901 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.770867 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-kube-api-access-rthkl\" (UniqueName: \"kubernetes.io/projected/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-kube-api-access-rthkl\") pod \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " Oct 30 12:26:09.770966 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.770930 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-credential-socket\") pod \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " Oct 30 12:26:09.771023 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.771005 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istiod-ca-cert\") pod \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " Oct 30 12:26:09.771163 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.771090 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-podinfo\") pod \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\" (UID: \"53efa6ea-7b24-4df7-b9b3-46ab86493fc1\") " Oct 30 12:26:09.771163 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.771097 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-data" pod "53efa6ea-7b24-4df7-b9b3-46ab86493fc1" (UID: "53efa6ea-7b24-4df7-b9b3-46ab86493fc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:09.771163 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.771062 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-workload-certs" pod "53efa6ea-7b24-4df7-b9b3-46ab86493fc1" (UID: "53efa6ea-7b24-4df7-b9b3-46ab86493fc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:09.771971 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.771880 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-credential-socket" pod "53efa6ea-7b24-4df7-b9b3-46ab86493fc1" (UID: "53efa6ea-7b24-4df7-b9b3-46ab86493fc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:09.772159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.772096 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.772308 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.772293 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.772411 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.772398 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.772483 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.772429 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istiod-ca-cert" pod "53efa6ea-7b24-4df7-b9b3-46ab86493fc1" (UID: "53efa6ea-7b24-4df7-b9b3-46ab86493fc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:26:09.774437 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.774409 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-envoy" pod "53efa6ea-7b24-4df7-b9b3-46ab86493fc1" (UID: "53efa6ea-7b24-4df7-b9b3-46ab86493fc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:09.774568 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.774423 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-podinfo" pod "53efa6ea-7b24-4df7-b9b3-46ab86493fc1" (UID: "53efa6ea-7b24-4df7-b9b3-46ab86493fc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:26:09.774717 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.774704 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-token" pod "53efa6ea-7b24-4df7-b9b3-46ab86493fc1" (UID: "53efa6ea-7b24-4df7-b9b3-46ab86493fc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:26:09.775156 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.775135 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-kube-api-access-rthkl" pod "53efa6ea-7b24-4df7-b9b3-46ab86493fc1" (UID: "53efa6ea-7b24-4df7-b9b3-46ab86493fc1"). InnerVolumeSpecName "kube-api-access-rthkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:26:09.873926 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.873829 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rthkl\" (UniqueName: \"kubernetes.io/projected/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-kube-api-access-rthkl\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.873926 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.873867 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.873926 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.873878 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.873926 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.873887 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.873926 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.873896 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.873926 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.873903 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53efa6ea-7b24-4df7-b9b3-46ab86493fc1-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.899500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.899472 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 12:26:09.899656 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.899522 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" podUID="df475a56-3557-40c3-84bb-0e812127ae21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 12:26:09.939248 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.939205 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" event={"ID":"1d5f9e36-e943-405f-9f53-6ffb08d31c67","Type":"ContainerDied","Data":"5bd0df2ad7d91628adc626180a351c50c0105e65f55437af0e794590526f2c20"} Oct 30 12:26:09.939702 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.939308 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" event={"ID":"1d5f9e36-e943-405f-9f53-6ffb08d31c67","Type":"ContainerDied","Data":"94e3b3fcd127465e6beb925a006f71826a9e12f51845fdf98d47336fce1bf50b"} Oct 30 12:26:09.940765 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.940736 4468 generic.go:356] "Generic (PLEG): container finished" podID="53efa6ea-7b24-4df7-b9b3-46ab86493fc1" containerID="675403ae1d4426df2a949bddfa0468e39be553afe17710d97c22057e5ba0b1f8" exitCode=0 Oct 30 12:26:09.940918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.940786 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" Oct 30 12:26:09.940918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.940806 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" event={"ID":"53efa6ea-7b24-4df7-b9b3-46ab86493fc1","Type":"ContainerDied","Data":"675403ae1d4426df2a949bddfa0468e39be553afe17710d97c22057e5ba0b1f8"} Oct 30 12:26:09.940918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.940871 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs" event={"ID":"53efa6ea-7b24-4df7-b9b3-46ab86493fc1","Type":"ContainerDied","Data":"a55b939c7a0180bde9fb2b7519c6d077c94332e5167835c39e710b8f95ce30f0"} Oct 30 12:26:09.940918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.940911 4468 scope.go:122] "RemoveContainer" containerID="675403ae1d4426df2a949bddfa0468e39be553afe17710d97c22057e5ba0b1f8" Oct 30 12:26:09.941294 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.941273 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml" Oct 30 12:26:09.954193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.954167 4468 scope.go:122] "RemoveContainer" containerID="675403ae1d4426df2a949bddfa0468e39be553afe17710d97c22057e5ba0b1f8" Oct 30 12:26:09.954555 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:26:09.954534 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"675403ae1d4426df2a949bddfa0468e39be553afe17710d97c22057e5ba0b1f8\": container with ID starting with 675403ae1d4426df2a949bddfa0468e39be553afe17710d97c22057e5ba0b1f8 not found: ID does not exist" containerID="675403ae1d4426df2a949bddfa0468e39be553afe17710d97c22057e5ba0b1f8" Oct 30 12:26:09.954607 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:09.954570 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"675403ae1d4426df2a949bddfa0468e39be553afe17710d97c22057e5ba0b1f8"} err="failed to get container status \"675403ae1d4426df2a949bddfa0468e39be553afe17710d97c22057e5ba0b1f8\": rpc error: code = NotFound desc = could not find container \"675403ae1d4426df2a949bddfa0468e39be553afe17710d97c22057e5ba0b1f8\": container with ID starting with 675403ae1d4426df2a949bddfa0468e39be553afe17710d97c22057e5ba0b1f8 not found: ID does not exist" Oct 30 12:26:10.000754 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:10.000710 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs"] Oct 30 12:26:10.025097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:10.023027 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-vcvxs"] Oct 30 12:26:10.083490 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:10.083461 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml"] Oct 30 12:26:10.088474 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:10.088444 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-nmpml"] Oct 30 12:26:10.695092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:10.687908 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn" Oct 30 12:26:10.900020 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:10.899980 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 12:26:10.900337 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:10.900301 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" podUID="df475a56-3557-40c3-84bb-0e812127ae21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 12:26:11.092211 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:11.092131 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d5f9e36-e943-405f-9f53-6ffb08d31c67" path="/var/lib/kubelet/pods/1d5f9e36-e943-405f-9f53-6ffb08d31c67/volumes" Oct 30 12:26:11.092608 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:11.092592 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53efa6ea-7b24-4df7-b9b3-46ab86493fc1" path="/var/lib/kubelet/pods/53efa6ea-7b24-4df7-b9b3-46ab86493fc1/volumes" Oct 30 12:26:11.900103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:11.900036 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 12:26:11.900266 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:11.900132 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" podUID="df475a56-3557-40c3-84bb-0e812127ae21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 12:26:12.904918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:12.904886 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:26:12.905435 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:12.905293 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:26:12.905965 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:12.905943 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:26:15.882096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:15.880517 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs"] Oct 30 12:26:15.910088 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:15.904870 4468 cpu_manager.go:414] "containerMap: removing container" podUID="1d5f9e36-e943-405f-9f53-6ffb08d31c67" containerName="istio-proxy" Oct 30 12:26:15.910088 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:15.904910 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d5f9e36-e943-405f-9f53-6ffb08d31c67" containerName="istio-proxy" Oct 30 12:26:15.910088 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:15.904977 4468 cpu_manager.go:414] "containerMap: removing container" podUID="53efa6ea-7b24-4df7-b9b3-46ab86493fc1" containerName="istio-proxy" Oct 30 12:26:15.910088 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:15.904986 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53efa6ea-7b24-4df7-b9b3-46ab86493fc1" containerName="istio-proxy" Oct 30 12:26:15.910088 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:15.905192 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d5f9e36-e943-405f-9f53-6ffb08d31c67" containerName="istio-proxy" Oct 30 12:26:15.910088 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:15.905215 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="53efa6ea-7b24-4df7-b9b3-46ab86493fc1" containerName="istio-proxy" Oct 30 12:26:15.917108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:15.915372 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs" Oct 30 12:26:15.928097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:15.926960 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs"] Oct 30 12:26:15.982716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:15.971145 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8e3112\"" Oct 30 12:26:15.982716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:15.971247 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8e3112-dockercfg-6jh9f\"" Oct 30 12:26:16.026603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:16.026562 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r"] Oct 30 12:26:16.041644 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:16.032114 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r" Oct 30 12:26:16.058438 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:16.058327 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r"] Oct 30 12:26:16.099849 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:16.099801 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rbgp\" (UniqueName: \"kubernetes.io/projected/38ad22b1-0685-49be-912b-5e72e094cb18-kube-api-access-7rbgp\") pod \"payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs\" (UID: \"38ad22b1-0685-49be-912b-5e72e094cb18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs" Oct 30 12:26:16.100411 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:16.100370 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ad22b1-0685-49be-912b-5e72e094cb18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs\" (UID: \"38ad22b1-0685-49be-912b-5e72e094cb18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs" Oct 30 12:26:16.202183 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:16.202149 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc9eb17-6f1d-4246-9849-9a0d75022e41-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r\" (UID: \"7dc9eb17-6f1d-4246-9849-9a0d75022e41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r" Oct 30 12:26:16.202349 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:16.202194 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ad22b1-0685-49be-912b-5e72e094cb18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs\" (UID: \"38ad22b1-0685-49be-912b-5e72e094cb18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs" Oct 30 12:26:16.202349 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:16.202316 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4q6p\" (UniqueName: \"kubernetes.io/projected/7dc9eb17-6f1d-4246-9849-9a0d75022e41-kube-api-access-b4q6p\") pod \"payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r\" (UID: \"7dc9eb17-6f1d-4246-9849-9a0d75022e41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r" Oct 30 12:26:16.202527 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:16.202512 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rbgp\" (UniqueName: \"kubernetes.io/projected/38ad22b1-0685-49be-912b-5e72e094cb18-kube-api-access-7rbgp\") pod \"payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs\" (UID: \"38ad22b1-0685-49be-912b-5e72e094cb18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs" Oct 30 12:26:16.202956 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:16.202938 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ad22b1-0685-49be-912b-5e72e094cb18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs\" (UID: \"38ad22b1-0685-49be-912b-5e72e094cb18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs" Oct 30 12:26:16.227753 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:16.227721 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rbgp\" (UniqueName: \"kubernetes.io/projected/38ad22b1-0685-49be-912b-5e72e094cb18-kube-api-access-7rbgp\") pod \"payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs\" (UID: \"38ad22b1-0685-49be-912b-5e72e094cb18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs" Oct 30 12:26:16.304162 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:16.304117 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc9eb17-6f1d-4246-9849-9a0d75022e41-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r\" (UID: \"7dc9eb17-6f1d-4246-9849-9a0d75022e41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r" Oct 30 12:26:16.304383 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:16.304246 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4q6p\" (UniqueName: \"kubernetes.io/projected/7dc9eb17-6f1d-4246-9849-9a0d75022e41-kube-api-access-b4q6p\") pod \"payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r\" (UID: \"7dc9eb17-6f1d-4246-9849-9a0d75022e41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r" Oct 30 12:26:16.304889 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:16.304860 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc9eb17-6f1d-4246-9849-9a0d75022e41-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r\" (UID: \"7dc9eb17-6f1d-4246-9849-9a0d75022e41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r" Oct 30 12:26:16.329186 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:16.329153 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4q6p\" (UniqueName: \"kubernetes.io/projected/7dc9eb17-6f1d-4246-9849-9a0d75022e41-kube-api-access-b4q6p\") pod \"payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r\" (UID: \"7dc9eb17-6f1d-4246-9849-9a0d75022e41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r" Oct 30 12:26:16.334970 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:16.334938 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs" Oct 30 12:26:16.374327 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:16.374286 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r" Oct 30 12:26:16.681788 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:26:16.681754 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ad22b1_0685_49be_912b_5e72e094cb18.slice/crio-c22482ae8d333d62a26e8afff62859d1230a70a510bf532ec63a0b00664ff462 WatchSource:0}: Error finding container c22482ae8d333d62a26e8afff62859d1230a70a510bf532ec63a0b00664ff462: Status 404 returned error can't find the container with id c22482ae8d333d62a26e8afff62859d1230a70a510bf532ec63a0b00664ff462 Oct 30 12:26:16.684243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:16.684217 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs"] Oct 30 12:26:16.718612 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:26:16.718582 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc9eb17_6f1d_4246_9849_9a0d75022e41.slice/crio-d76b4c9dd4afa1920ef84f103482548b9d552ba6b805dbc2b9e840f3dac1d365 WatchSource:0}: Error finding container d76b4c9dd4afa1920ef84f103482548b9d552ba6b805dbc2b9e840f3dac1d365: Status 404 returned error can't find the container with id d76b4c9dd4afa1920ef84f103482548b9d552ba6b805dbc2b9e840f3dac1d365 Oct 30 12:26:16.720283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:16.720260 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r"] Oct 30 12:26:17.030196 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:17.030085 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r" event={"ID":"7dc9eb17-6f1d-4246-9849-9a0d75022e41","Type":"ContainerStarted","Data":"ddb15156f00bbabf77f9d040a64ecd212e81223ffac43d1ab330b28f65c47394"} Oct 30 12:26:17.030196 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:17.030161 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r" Oct 30 12:26:17.030196 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:17.030183 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r" event={"ID":"7dc9eb17-6f1d-4246-9849-9a0d75022e41","Type":"ContainerStarted","Data":"d76b4c9dd4afa1920ef84f103482548b9d552ba6b805dbc2b9e840f3dac1d365"} Oct 30 12:26:17.032164 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:17.032136 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs" event={"ID":"38ad22b1-0685-49be-912b-5e72e094cb18","Type":"ContainerStarted","Data":"567ed6e1844117b36a5925ada2b37dd908c508e85a0cd3545cf4b18bec12b585"} Oct 30 12:26:17.032306 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:17.032181 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs" event={"ID":"38ad22b1-0685-49be-912b-5e72e094cb18","Type":"ContainerStarted","Data":"c22482ae8d333d62a26e8afff62859d1230a70a510bf532ec63a0b00664ff462"} Oct 30 12:26:17.032367 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:17.032301 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs" Oct 30 12:26:17.076246 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:17.076194 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r" podStartSLOduration=2.076176798 podStartE2EDuration="2.076176798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:17.073570794 +0000 UTC m=+1928.107709191" watchObservedRunningTime="2026-10-30 12:26:17.076176798 +0000 UTC m=+1928.110315205" Oct 30 12:26:17.121013 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:17.120953 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs" podStartSLOduration=2.120937367 podStartE2EDuration="2.120937367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:17.116594696 +0000 UTC m=+1928.150733092" watchObservedRunningTime="2026-10-30 12:26:17.120937367 +0000 UTC m=+1928.155075762" Oct 30 12:26:19.944718 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:19.944686 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6" Oct 30 12:26:19.945198 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:19.944766 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx" Oct 30 12:26:19.945198 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:19.944810 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47" Oct 30 12:26:23.600674 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:26:23.600583 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:26:28.042044 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:28.042008 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r" Oct 30 12:26:28.042497 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:28.042121 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs" Oct 30 12:26:31.519766 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:31.519725 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl"] Oct 30 12:26:31.525942 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:31.525914 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl" Oct 30 12:26:31.544352 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:31.544318 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl"] Oct 30 12:26:31.551172 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:31.551113 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-096834\"" Oct 30 12:26:31.554314 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:31.554288 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-096834-dockercfg-dgz4c\"" Oct 30 12:26:31.647171 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:31.623345 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d065fa-815a-4f94-8a46-17f9a3de8368-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl\" (UID: \"01d065fa-815a-4f94-8a46-17f9a3de8368\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl" Oct 30 12:26:31.647171 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:31.623416 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88q4m\" (UniqueName: \"kubernetes.io/projected/01d065fa-815a-4f94-8a46-17f9a3de8368-kube-api-access-88q4m\") pod \"payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl\" (UID: \"01d065fa-815a-4f94-8a46-17f9a3de8368\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl" Oct 30 12:26:31.707282 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:31.707237 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8"] Oct 30 12:26:31.758345 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:31.726226 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d065fa-815a-4f94-8a46-17f9a3de8368-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl\" (UID: \"01d065fa-815a-4f94-8a46-17f9a3de8368\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl" Oct 30 12:26:31.762841 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:31.762806 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88q4m\" (UniqueName: \"kubernetes.io/projected/01d065fa-815a-4f94-8a46-17f9a3de8368-kube-api-access-88q4m\") pod \"payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl\" (UID: \"01d065fa-815a-4f94-8a46-17f9a3de8368\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl" Oct 30 12:26:31.764345 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:31.753330 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d065fa-815a-4f94-8a46-17f9a3de8368-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl\" (UID: \"01d065fa-815a-4f94-8a46-17f9a3de8368\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl" Oct 30 12:26:31.818608 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:31.818506 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8"] Oct 30 12:26:31.819005 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:31.818985 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8" Oct 30 12:26:31.835117 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:31.823966 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88q4m\" (UniqueName: \"kubernetes.io/projected/01d065fa-815a-4f94-8a46-17f9a3de8368-kube-api-access-88q4m\") pod \"payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl\" (UID: \"01d065fa-815a-4f94-8a46-17f9a3de8368\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl" Oct 30 12:26:31.844537 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:31.844162 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl" Oct 30 12:26:31.968047 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:31.967844 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mldmk\" (UniqueName: \"kubernetes.io/projected/28a586ed-6c97-48ed-b147-d197fa84ab91-kube-api-access-mldmk\") pod \"payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8\" (UID: \"28a586ed-6c97-48ed-b147-d197fa84ab91\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8" Oct 30 12:26:31.968831 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:31.968647 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a586ed-6c97-48ed-b147-d197fa84ab91-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8\" (UID: \"28a586ed-6c97-48ed-b147-d197fa84ab91\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8" Oct 30 12:26:32.070495 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:32.070387 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a586ed-6c97-48ed-b147-d197fa84ab91-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8\" (UID: \"28a586ed-6c97-48ed-b147-d197fa84ab91\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8" Oct 30 12:26:32.070679 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:32.070563 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mldmk\" (UniqueName: \"kubernetes.io/projected/28a586ed-6c97-48ed-b147-d197fa84ab91-kube-api-access-mldmk\") pod \"payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8\" (UID: \"28a586ed-6c97-48ed-b147-d197fa84ab91\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8" Oct 30 12:26:32.071469 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:32.071440 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a586ed-6c97-48ed-b147-d197fa84ab91-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8\" (UID: \"28a586ed-6c97-48ed-b147-d197fa84ab91\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8" Oct 30 12:26:32.103714 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:32.103676 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mldmk\" (UniqueName: \"kubernetes.io/projected/28a586ed-6c97-48ed-b147-d197fa84ab91-kube-api-access-mldmk\") pod \"payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8\" (UID: \"28a586ed-6c97-48ed-b147-d197fa84ab91\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8" Oct 30 12:26:32.185803 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:26:32.185767 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d065fa_815a_4f94_8a46_17f9a3de8368.slice/crio-fb0f9d431ef03ee3aa38f86695f7b02df5bb3410415bf9d10d4b549fcf676570 WatchSource:0}: Error finding container fb0f9d431ef03ee3aa38f86695f7b02df5bb3410415bf9d10d4b549fcf676570: Status 404 returned error can't find the container with id fb0f9d431ef03ee3aa38f86695f7b02df5bb3410415bf9d10d4b549fcf676570 Oct 30 12:26:32.188791 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:32.188762 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl"] Oct 30 12:26:32.193649 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:32.193625 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8" Oct 30 12:26:32.370542 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:32.357386 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl" event={"ID":"01d065fa-815a-4f94-8a46-17f9a3de8368","Type":"ContainerStarted","Data":"fb0f9d431ef03ee3aa38f86695f7b02df5bb3410415bf9d10d4b549fcf676570"} Oct 30 12:26:32.664602 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:26:32.664570 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a586ed_6c97_48ed_b147_d197fa84ab91.slice/crio-4154d78ebc5f97b2ca1de431ce275de11b38f61449a825ceb236c36cf4b0e56b WatchSource:0}: Error finding container 4154d78ebc5f97b2ca1de431ce275de11b38f61449a825ceb236c36cf4b0e56b: Status 404 returned error can't find the container with id 4154d78ebc5f97b2ca1de431ce275de11b38f61449a825ceb236c36cf4b0e56b Oct 30 12:26:32.668594 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:32.668432 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8"] Oct 30 12:26:33.367265 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:33.367225 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8" event={"ID":"28a586ed-6c97-48ed-b147-d197fa84ab91","Type":"ContainerStarted","Data":"a401e7c18e8cf9b7e8442832b64ecd5d10df2907dd32b384be0221ff2435a52c"} Oct 30 12:26:33.367479 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:33.367287 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8" event={"ID":"28a586ed-6c97-48ed-b147-d197fa84ab91","Type":"ContainerStarted","Data":"4154d78ebc5f97b2ca1de431ce275de11b38f61449a825ceb236c36cf4b0e56b"} Oct 30 12:26:33.367479 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:33.367443 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8" Oct 30 12:26:33.368964 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:33.368937 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl" event={"ID":"01d065fa-815a-4f94-8a46-17f9a3de8368","Type":"ContainerStarted","Data":"1be417ef86cfea9f4289300e6d762a3ef57240c776893e0e2d65d849aff69480"} Oct 30 12:26:33.369080 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:33.368991 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl" Oct 30 12:26:33.411015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:33.410948 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8" podStartSLOduration=2.410934447 podStartE2EDuration="2.410934447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:33.409944471 +0000 UTC m=+1944.444082868" watchObservedRunningTime="2026-10-30 12:26:33.410934447 +0000 UTC m=+1944.445072842" Oct 30 12:26:33.453273 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:33.453218 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl" podStartSLOduration=2.453203846 podStartE2EDuration="2.453203846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:33.452767473 +0000 UTC m=+1944.486905869" watchObservedRunningTime="2026-10-30 12:26:33.453203846 +0000 UTC m=+1944.487342242" Oct 30 12:26:37.459488 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:37.459452 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554"] Oct 30 12:26:37.464380 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:37.464346 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554" Oct 30 12:26:37.483008 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:37.482965 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554"] Oct 30 12:26:37.491577 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:37.491544 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5c57af\"" Oct 30 12:26:37.491856 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:37.491836 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5c57af-dockercfg-hg5dn\"" Oct 30 12:26:37.513879 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:37.513840 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvnpd\" (UniqueName: \"kubernetes.io/projected/c9f1a670-d9c8-4c44-bec0-10c84cf880f4-kube-api-access-pvnpd\") pod \"payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554\" (UID: \"c9f1a670-d9c8-4c44-bec0-10c84cf880f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554" Oct 30 12:26:37.514137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:37.513911 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f1a670-d9c8-4c44-bec0-10c84cf880f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554\" (UID: \"c9f1a670-d9c8-4c44-bec0-10c84cf880f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554" Oct 30 12:26:37.631091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:37.630274 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvnpd\" (UniqueName: \"kubernetes.io/projected/c9f1a670-d9c8-4c44-bec0-10c84cf880f4-kube-api-access-pvnpd\") pod \"payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554\" (UID: \"c9f1a670-d9c8-4c44-bec0-10c84cf880f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554" Oct 30 12:26:37.631091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:37.630360 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f1a670-d9c8-4c44-bec0-10c84cf880f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554\" (UID: \"c9f1a670-d9c8-4c44-bec0-10c84cf880f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554" Oct 30 12:26:37.635095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:37.631593 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f1a670-d9c8-4c44-bec0-10c84cf880f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554\" (UID: \"c9f1a670-d9c8-4c44-bec0-10c84cf880f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554" Oct 30 12:26:37.670644 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:37.666858 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvnpd\" (UniqueName: \"kubernetes.io/projected/c9f1a670-d9c8-4c44-bec0-10c84cf880f4-kube-api-access-pvnpd\") pod \"payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554\" (UID: \"c9f1a670-d9c8-4c44-bec0-10c84cf880f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554" Oct 30 12:26:37.781035 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:37.780222 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554" Oct 30 12:26:38.105231 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:26:38.105197 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f1a670_d9c8_4c44_bec0_10c84cf880f4.slice/crio-6c9bc27d28ceecf91d116125e7fca5ce3ad73d07af629f50754ace78e8917643 WatchSource:0}: Error finding container 6c9bc27d28ceecf91d116125e7fca5ce3ad73d07af629f50754ace78e8917643: Status 404 returned error can't find the container with id 6c9bc27d28ceecf91d116125e7fca5ce3ad73d07af629f50754ace78e8917643 Oct 30 12:26:38.109243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:38.109219 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554"] Oct 30 12:26:38.406731 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:38.406704 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554" event={"ID":"c9f1a670-d9c8-4c44-bec0-10c84cf880f4","Type":"ContainerStarted","Data":"4db5015972d35d8334280b6a8efc264148cd1ea8fca166d0fee576a5686dc1e3"} Oct 30 12:26:38.406933 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:38.406758 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554" event={"ID":"c9f1a670-d9c8-4c44-bec0-10c84cf880f4","Type":"ContainerStarted","Data":"6c9bc27d28ceecf91d116125e7fca5ce3ad73d07af629f50754ace78e8917643"} Oct 30 12:26:38.406933 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:38.406780 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554" Oct 30 12:26:38.451130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:38.451065 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554" podStartSLOduration=1.451052011 podStartE2EDuration="1.451052011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:38.449649815 +0000 UTC m=+1949.483788212" watchObservedRunningTime="2026-10-30 12:26:38.451052011 +0000 UTC m=+1949.485190416" Oct 30 12:26:41.244158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:41.244115 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv"] Oct 30 12:26:44.379859 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:44.379822 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8" Oct 30 12:26:44.380452 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:44.380289 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl" Oct 30 12:26:46.245535 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:46.245487 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv" podUID="d109c257-63ad-4bc4-a8f8-4877db8be90e" containerName="payload-processing" containerID="cri-o://cd5787c3174b7d67ac0d119cf28aeb856ab7582fe9c99bc03b0da37996373e48" gracePeriod=25 Oct 30 12:26:46.480710 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:46.480670 4468 generic.go:356] "Generic (PLEG): container finished" podID="d109c257-63ad-4bc4-a8f8-4877db8be90e" containerID="cd5787c3174b7d67ac0d119cf28aeb856ab7582fe9c99bc03b0da37996373e48" exitCode=0 Oct 30 12:26:46.480925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:46.480739 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv" event={"ID":"d109c257-63ad-4bc4-a8f8-4877db8be90e","Type":"ContainerDied","Data":"cd5787c3174b7d67ac0d119cf28aeb856ab7582fe9c99bc03b0da37996373e48"} Oct 30 12:26:46.523750 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:46.523724 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv" Oct 30 12:26:46.656022 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:46.655988 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d109c257-63ad-4bc4-a8f8-4877db8be90e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d109c257-63ad-4bc4-a8f8-4877db8be90e-plugins-config-volume\") pod \"d109c257-63ad-4bc4-a8f8-4877db8be90e\" (UID: \"d109c257-63ad-4bc4-a8f8-4877db8be90e\") " Oct 30 12:26:46.656224 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:46.656103 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d109c257-63ad-4bc4-a8f8-4877db8be90e-kube-api-access-llf4k\" (UniqueName: \"kubernetes.io/projected/d109c257-63ad-4bc4-a8f8-4877db8be90e-kube-api-access-llf4k\") pod \"d109c257-63ad-4bc4-a8f8-4877db8be90e\" (UID: \"d109c257-63ad-4bc4-a8f8-4877db8be90e\") " Oct 30 12:26:46.656380 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:46.656354 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d109c257-63ad-4bc4-a8f8-4877db8be90e-plugins-config-volume" pod "d109c257-63ad-4bc4-a8f8-4877db8be90e" (UID: "d109c257-63ad-4bc4-a8f8-4877db8be90e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:26:46.656634 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:46.656620 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d109c257-63ad-4bc4-a8f8-4877db8be90e-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:46.658282 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:46.658252 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d109c257-63ad-4bc4-a8f8-4877db8be90e-kube-api-access-llf4k" pod "d109c257-63ad-4bc4-a8f8-4877db8be90e" (UID: "d109c257-63ad-4bc4-a8f8-4877db8be90e"). InnerVolumeSpecName "kube-api-access-llf4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:26:46.757850 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:46.757750 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llf4k\" (UniqueName: \"kubernetes.io/projected/d109c257-63ad-4bc4-a8f8-4877db8be90e-kube-api-access-llf4k\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:47.539133 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:47.532896 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv" event={"ID":"d109c257-63ad-4bc4-a8f8-4877db8be90e","Type":"ContainerDied","Data":"11d194e35ab648a0b91484b531dc65f6c3464300fe40891b180eeaeb84b056ae"} Oct 30 12:26:47.539133 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:47.533019 4468 scope.go:122] "RemoveContainer" containerID="cd5787c3174b7d67ac0d119cf28aeb856ab7582fe9c99bc03b0da37996373e48" Oct 30 12:26:47.539133 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:47.533545 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv" Oct 30 12:26:47.604950 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:47.604917 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv"] Oct 30 12:26:47.609787 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:47.609754 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aef61e-79dc689d48-558wv"] Oct 30 12:26:49.093141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:49.093098 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d109c257-63ad-4bc4-a8f8-4877db8be90e" path="/var/lib/kubelet/pods/d109c257-63ad-4bc4-a8f8-4877db8be90e/volumes" Oct 30 12:26:49.418778 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:49.418747 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554" Oct 30 12:26:51.545183 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:51.545144 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn"] Oct 30 12:26:53.609620 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:26:53.609583 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:26:56.545617 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:56.545540 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn" podUID="58c942ca-8c8b-4e96-b2ed-e2cbe45af71c" containerName="payload-pre-processing" containerID="cri-o://aa7835a289d03145b929ef0bc6c231da5383806d8e4d56f60a2bde2969f24f69" gracePeriod=25 Oct 30 12:26:56.848505 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:56.848473 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn" Oct 30 12:26:56.954180 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:56.954142 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c942ca-8c8b-4e96-b2ed-e2cbe45af71c-kube-api-access-9xfzt\" (UniqueName: \"kubernetes.io/projected/58c942ca-8c8b-4e96-b2ed-e2cbe45af71c-kube-api-access-9xfzt\") pod \"58c942ca-8c8b-4e96-b2ed-e2cbe45af71c\" (UID: \"58c942ca-8c8b-4e96-b2ed-e2cbe45af71c\") " Oct 30 12:26:56.954356 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:56.954202 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58c942ca-8c8b-4e96-b2ed-e2cbe45af71c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c942ca-8c8b-4e96-b2ed-e2cbe45af71c-plugins-config-volume\") pod \"58c942ca-8c8b-4e96-b2ed-e2cbe45af71c\" (UID: \"58c942ca-8c8b-4e96-b2ed-e2cbe45af71c\") " Oct 30 12:26:56.954599 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:56.954569 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c942ca-8c8b-4e96-b2ed-e2cbe45af71c-plugins-config-volume" pod "58c942ca-8c8b-4e96-b2ed-e2cbe45af71c" (UID: "58c942ca-8c8b-4e96-b2ed-e2cbe45af71c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:26:56.954722 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:56.954682 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c942ca-8c8b-4e96-b2ed-e2cbe45af71c-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:56.956340 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:56.956318 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c942ca-8c8b-4e96-b2ed-e2cbe45af71c-kube-api-access-9xfzt" pod "58c942ca-8c8b-4e96-b2ed-e2cbe45af71c" (UID: "58c942ca-8c8b-4e96-b2ed-e2cbe45af71c"). InnerVolumeSpecName "kube-api-access-9xfzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:26:57.055663 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:57.055623 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xfzt\" (UniqueName: \"kubernetes.io/projected/58c942ca-8c8b-4e96-b2ed-e2cbe45af71c-kube-api-access-9xfzt\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:26:57.627103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:57.627029 4468 generic.go:356] "Generic (PLEG): container finished" podID="58c942ca-8c8b-4e96-b2ed-e2cbe45af71c" containerID="aa7835a289d03145b929ef0bc6c231da5383806d8e4d56f60a2bde2969f24f69" exitCode=0 Oct 30 12:26:57.627567 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:57.627107 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn" Oct 30 12:26:57.627567 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:57.627102 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn" event={"ID":"58c942ca-8c8b-4e96-b2ed-e2cbe45af71c","Type":"ContainerDied","Data":"aa7835a289d03145b929ef0bc6c231da5383806d8e4d56f60a2bde2969f24f69"} Oct 30 12:26:57.627567 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:57.627224 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn" event={"ID":"58c942ca-8c8b-4e96-b2ed-e2cbe45af71c","Type":"ContainerDied","Data":"52f7a718a9bfcfd705857fac3283ecbe9f662c3e8dbd09f6e92120bb89687ac1"} Oct 30 12:26:57.627567 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:57.627246 4468 scope.go:122] "RemoveContainer" containerID="aa7835a289d03145b929ef0bc6c231da5383806d8e4d56f60a2bde2969f24f69" Oct 30 12:26:57.637901 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:57.637881 4468 scope.go:122] "RemoveContainer" containerID="aa7835a289d03145b929ef0bc6c231da5383806d8e4d56f60a2bde2969f24f69" Oct 30 12:26:57.638278 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:26:57.638238 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7835a289d03145b929ef0bc6c231da5383806d8e4d56f60a2bde2969f24f69\": container with ID starting with aa7835a289d03145b929ef0bc6c231da5383806d8e4d56f60a2bde2969f24f69 not found: ID does not exist" containerID="aa7835a289d03145b929ef0bc6c231da5383806d8e4d56f60a2bde2969f24f69" Oct 30 12:26:57.638359 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:57.638287 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7835a289d03145b929ef0bc6c231da5383806d8e4d56f60a2bde2969f24f69"} err="failed to get container status \"aa7835a289d03145b929ef0bc6c231da5383806d8e4d56f60a2bde2969f24f69\": rpc error: code = NotFound desc = could not find container \"aa7835a289d03145b929ef0bc6c231da5383806d8e4d56f60a2bde2969f24f69\": container with ID starting with aa7835a289d03145b929ef0bc6c231da5383806d8e4d56f60a2bde2969f24f69 not found: ID does not exist" Oct 30 12:26:57.688126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:57.688090 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn"] Oct 30 12:26:57.701647 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:57.701612 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-54bb7dc7f9-pt8bn"] Oct 30 12:26:59.091007 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:26:59.090975 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58c942ca-8c8b-4e96-b2ed-e2cbe45af71c" path="/var/lib/kubelet/pods/58c942ca-8c8b-4e96-b2ed-e2cbe45af71c/volumes" Oct 30 12:27:12.504247 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:12.504210 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj"] Oct 30 12:27:12.506233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:12.506200 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" podUID="6aed0019-79e9-4471-97b9-830763499da0" containerName="maas-api" containerID="cri-o://73dc006e959a8f5aea45ab4b7fc38f8d0e7823acb98142adb09fff073879ce62" gracePeriod=30 Oct 30 12:27:12.985002 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:12.984964 4468 generic.go:356] "Generic (PLEG): container finished" podID="6aed0019-79e9-4471-97b9-830763499da0" containerID="73dc006e959a8f5aea45ab4b7fc38f8d0e7823acb98142adb09fff073879ce62" exitCode=0 Oct 30 12:27:12.985893 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:12.985520 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" event={"ID":"6aed0019-79e9-4471-97b9-830763499da0","Type":"ContainerDied","Data":"73dc006e959a8f5aea45ab4b7fc38f8d0e7823acb98142adb09fff073879ce62"} Oct 30 12:27:13.101482 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:13.101449 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" Oct 30 12:27:13.291108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:13.290991 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aed0019-79e9-4471-97b9-830763499da0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aed0019-79e9-4471-97b9-830763499da0-metrics-certs\") pod \"6aed0019-79e9-4471-97b9-830763499da0\" (UID: \"6aed0019-79e9-4471-97b9-830763499da0\") " Oct 30 12:27:13.291282 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:13.291267 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aed0019-79e9-4471-97b9-830763499da0-kube-api-access-h62qf\" (UniqueName: \"kubernetes.io/projected/6aed0019-79e9-4471-97b9-830763499da0-kube-api-access-h62qf\") pod \"6aed0019-79e9-4471-97b9-830763499da0\" (UID: \"6aed0019-79e9-4471-97b9-830763499da0\") " Oct 30 12:27:13.291341 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:13.291306 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aed0019-79e9-4471-97b9-830763499da0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aed0019-79e9-4471-97b9-830763499da0-openshift-service-ca\") pod \"6aed0019-79e9-4471-97b9-830763499da0\" (UID: \"6aed0019-79e9-4471-97b9-830763499da0\") " Oct 30 12:27:13.291397 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:13.291353 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aed0019-79e9-4471-97b9-830763499da0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6aed0019-79e9-4471-97b9-830763499da0-maas-api-tls\") pod \"6aed0019-79e9-4471-97b9-830763499da0\" (UID: \"6aed0019-79e9-4471-97b9-830763499da0\") " Oct 30 12:27:13.291700 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:13.291671 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aed0019-79e9-4471-97b9-830763499da0-openshift-service-ca" pod "6aed0019-79e9-4471-97b9-830763499da0" (UID: "6aed0019-79e9-4471-97b9-830763499da0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:27:13.292016 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:13.291994 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aed0019-79e9-4471-97b9-830763499da0-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:27:13.293419 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:13.293398 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aed0019-79e9-4471-97b9-830763499da0-metrics-certs" pod "6aed0019-79e9-4471-97b9-830763499da0" (UID: "6aed0019-79e9-4471-97b9-830763499da0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:27:13.293506 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:13.293429 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aed0019-79e9-4471-97b9-830763499da0-maas-api-tls" pod "6aed0019-79e9-4471-97b9-830763499da0" (UID: "6aed0019-79e9-4471-97b9-830763499da0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:27:13.293506 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:13.293433 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aed0019-79e9-4471-97b9-830763499da0-kube-api-access-h62qf" pod "6aed0019-79e9-4471-97b9-830763499da0" (UID: "6aed0019-79e9-4471-97b9-830763499da0"). InnerVolumeSpecName "kube-api-access-h62qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:13.395227 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:13.394622 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aed0019-79e9-4471-97b9-830763499da0-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:27:13.395227 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:13.394754 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h62qf\" (UniqueName: \"kubernetes.io/projected/6aed0019-79e9-4471-97b9-830763499da0-kube-api-access-h62qf\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:27:13.395227 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:13.394773 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6aed0019-79e9-4471-97b9-830763499da0-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:27:13.993576 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:13.993543 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" Oct 30 12:27:13.994062 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:13.993568 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj" event={"ID":"6aed0019-79e9-4471-97b9-830763499da0","Type":"ContainerDied","Data":"0868f8ec9f6ab4e26460f53ff0de035ecbea4686119e4cb41f33afc6782ec9c9"} Oct 30 12:27:13.994062 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:13.993663 4468 scope.go:122] "RemoveContainer" containerID="73dc006e959a8f5aea45ab4b7fc38f8d0e7823acb98142adb09fff073879ce62" Oct 30 12:27:14.059334 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:14.059296 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj"] Oct 30 12:27:14.072421 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:14.072386 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aef61e-55cc7cbc55-kwxcj"] Oct 30 12:27:15.091320 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:15.091289 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aed0019-79e9-4471-97b9-830763499da0" path="/var/lib/kubelet/pods/6aed0019-79e9-4471-97b9-830763499da0/volumes" Oct 30 12:27:17.168801 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.168767 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt"] Oct 30 12:27:17.169804 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.169774 4468 cpu_manager.go:414] "containerMap: removing container" podUID="6aed0019-79e9-4471-97b9-830763499da0" containerName="maas-api" Oct 30 12:27:17.169906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.169812 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aed0019-79e9-4471-97b9-830763499da0" containerName="maas-api" Oct 30 12:27:17.169906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.169851 4468 cpu_manager.go:414] "containerMap: removing container" podUID="d109c257-63ad-4bc4-a8f8-4877db8be90e" containerName="payload-processing" Oct 30 12:27:17.169906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.169859 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d109c257-63ad-4bc4-a8f8-4877db8be90e" containerName="payload-processing" Oct 30 12:27:17.169906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.169875 4468 cpu_manager.go:414] "containerMap: removing container" podUID="58c942ca-8c8b-4e96-b2ed-e2cbe45af71c" containerName="payload-pre-processing" Oct 30 12:27:17.169906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.169883 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c942ca-8c8b-4e96-b2ed-e2cbe45af71c" containerName="payload-pre-processing" Oct 30 12:27:17.170274 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.170136 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aed0019-79e9-4471-97b9-830763499da0" containerName="maas-api" Oct 30 12:27:17.170274 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.170156 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d109c257-63ad-4bc4-a8f8-4877db8be90e" containerName="payload-processing" Oct 30 12:27:17.170274 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.170168 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c942ca-8c8b-4e96-b2ed-e2cbe45af71c" containerName="payload-pre-processing" Oct 30 12:27:17.176040 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.176011 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt"] Oct 30 12:27:17.176197 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.176131 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt" Oct 30 12:27:17.176737 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.176715 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q"] Oct 30 12:27:17.181559 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.181535 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q" Oct 30 12:27:17.202346 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.202312 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q"] Oct 30 12:27:17.212342 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.212314 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-266486\"" Oct 30 12:27:17.212474 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.212314 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-266486-dockercfg-hs7cj\"" Oct 30 12:27:17.285950 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.285915 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbm4l\" (UniqueName: \"kubernetes.io/projected/5503f728-1761-4eba-905c-47d8e4bb6677-kube-api-access-zbm4l\") pod \"payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q\" (UID: \"5503f728-1761-4eba-905c-47d8e4bb6677\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q" Oct 30 12:27:17.286188 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.285961 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fafb0ac3-55ef-4d91-ba47-4ad9b392b988-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt\" (UID: \"fafb0ac3-55ef-4d91-ba47-4ad9b392b988\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt" Oct 30 12:27:17.286312 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.286292 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8zxc\" (UniqueName: \"kubernetes.io/projected/fafb0ac3-55ef-4d91-ba47-4ad9b392b988-kube-api-access-k8zxc\") pod \"payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt\" (UID: \"fafb0ac3-55ef-4d91-ba47-4ad9b392b988\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt" Oct 30 12:27:17.286406 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.286388 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5503f728-1761-4eba-905c-47d8e4bb6677-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q\" (UID: \"5503f728-1761-4eba-905c-47d8e4bb6677\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q" Oct 30 12:27:17.387593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.387554 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5503f728-1761-4eba-905c-47d8e4bb6677-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q\" (UID: \"5503f728-1761-4eba-905c-47d8e4bb6677\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q" Oct 30 12:27:17.387770 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.387727 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbm4l\" (UniqueName: \"kubernetes.io/projected/5503f728-1761-4eba-905c-47d8e4bb6677-kube-api-access-zbm4l\") pod \"payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q\" (UID: \"5503f728-1761-4eba-905c-47d8e4bb6677\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q" Oct 30 12:27:17.387770 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.387754 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fafb0ac3-55ef-4d91-ba47-4ad9b392b988-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt\" (UID: \"fafb0ac3-55ef-4d91-ba47-4ad9b392b988\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt" Oct 30 12:27:17.387895 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.387879 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8zxc\" (UniqueName: \"kubernetes.io/projected/fafb0ac3-55ef-4d91-ba47-4ad9b392b988-kube-api-access-k8zxc\") pod \"payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt\" (UID: \"fafb0ac3-55ef-4d91-ba47-4ad9b392b988\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt" Oct 30 12:27:17.388533 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.388504 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5503f728-1761-4eba-905c-47d8e4bb6677-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q\" (UID: \"5503f728-1761-4eba-905c-47d8e4bb6677\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q" Oct 30 12:27:17.388652 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.388635 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fafb0ac3-55ef-4d91-ba47-4ad9b392b988-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt\" (UID: \"fafb0ac3-55ef-4d91-ba47-4ad9b392b988\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt" Oct 30 12:27:17.416780 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.416746 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8zxc\" (UniqueName: \"kubernetes.io/projected/fafb0ac3-55ef-4d91-ba47-4ad9b392b988-kube-api-access-k8zxc\") pod \"payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt\" (UID: \"fafb0ac3-55ef-4d91-ba47-4ad9b392b988\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt" Oct 30 12:27:17.416952 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.416822 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbm4l\" (UniqueName: \"kubernetes.io/projected/5503f728-1761-4eba-905c-47d8e4bb6677-kube-api-access-zbm4l\") pod \"payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q\" (UID: \"5503f728-1761-4eba-905c-47d8e4bb6677\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q" Oct 30 12:27:17.507318 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.507228 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt" Oct 30 12:27:17.514502 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.514469 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q" Oct 30 12:27:17.860556 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:27:17.860517 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfafb0ac3_55ef_4d91_ba47_4ad9b392b988.slice/crio-1a1853daa3176afa2aaf4170f7c6c6b483875fc0326d0865545541385a16a736 WatchSource:0}: Error finding container 1a1853daa3176afa2aaf4170f7c6c6b483875fc0326d0865545541385a16a736: Status 404 returned error can't find the container with id 1a1853daa3176afa2aaf4170f7c6c6b483875fc0326d0865545541385a16a736 Oct 30 12:27:17.865956 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.865932 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt"] Oct 30 12:27:17.873394 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:27:17.873366 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5503f728_1761_4eba_905c_47d8e4bb6677.slice/crio-3133d659a338073a56bdff2d54bce868d47aea0bfebdd46d129e8048ff4f4d1c WatchSource:0}: Error finding container 3133d659a338073a56bdff2d54bce868d47aea0bfebdd46d129e8048ff4f4d1c: Status 404 returned error can't find the container with id 3133d659a338073a56bdff2d54bce868d47aea0bfebdd46d129e8048ff4f4d1c Oct 30 12:27:17.874114 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:17.874082 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q"] Oct 30 12:27:18.092786 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:18.092747 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q" event={"ID":"5503f728-1761-4eba-905c-47d8e4bb6677","Type":"ContainerStarted","Data":"35cf38cdaf3749007196d2100518aed9251b2456e218446df0ae504149178116"} Oct 30 12:27:18.092993 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:18.092801 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q" event={"ID":"5503f728-1761-4eba-905c-47d8e4bb6677","Type":"ContainerStarted","Data":"3133d659a338073a56bdff2d54bce868d47aea0bfebdd46d129e8048ff4f4d1c"} Oct 30 12:27:18.092993 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:18.092947 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q" Oct 30 12:27:18.094554 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:18.094533 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt" event={"ID":"fafb0ac3-55ef-4d91-ba47-4ad9b392b988","Type":"ContainerStarted","Data":"3624bcf3b0467397cb6e0d85953a203fbb5548c9e7cd392dd026e59ac11e2ca9"} Oct 30 12:27:18.094654 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:18.094570 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt" event={"ID":"fafb0ac3-55ef-4d91-ba47-4ad9b392b988","Type":"ContainerStarted","Data":"1a1853daa3176afa2aaf4170f7c6c6b483875fc0326d0865545541385a16a736"} Oct 30 12:27:18.094654 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:18.094602 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt" Oct 30 12:27:18.141057 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:18.140952 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q" podStartSLOduration=1.140931678 podStartE2EDuration="1.140931678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:27:18.140459773 +0000 UTC m=+1989.174598172" watchObservedRunningTime="2026-10-30 12:27:18.140931678 +0000 UTC m=+1989.175070074" Oct 30 12:27:18.189563 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:18.189495 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt" podStartSLOduration=2.189476295 podStartE2EDuration="2.189476295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:27:18.184522066 +0000 UTC m=+1989.218660462" watchObservedRunningTime="2026-10-30 12:27:18.189476295 +0000 UTC m=+1989.223614692" Oct 30 12:27:22.229801 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.229764 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r"] Oct 30 12:27:22.235786 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.235760 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.267934 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.267897 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r"] Oct 30 12:27:22.271193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.271169 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7eee3b-openshift-default-dockercfg-26l9x\"" Oct 30 12:27:22.313580 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.313545 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-data\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.313768 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.313599 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-envoy\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.313768 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.313729 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-workload-socket\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.313768 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.313759 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-workload-certs\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.313916 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.313782 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3acebec4-84f8-415a-84f7-f8f7d9391a85-istiod-ca-cert\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.313916 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.313841 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-credential-socket\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.313916 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.313890 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-token\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.314182 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.313934 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-podinfo\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.314264 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.314238 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwkf4\" (UniqueName: \"kubernetes.io/projected/3acebec4-84f8-415a-84f7-f8f7d9391a85-kube-api-access-fwkf4\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.452615 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.452577 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwkf4\" (UniqueName: \"kubernetes.io/projected/3acebec4-84f8-415a-84f7-f8f7d9391a85-kube-api-access-fwkf4\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.453528 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.453501 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-data\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.454235 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.454212 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-envoy\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.472096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.454767 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-workload-socket\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.473650 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.473620 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-workload-certs\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.473830 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.473815 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3acebec4-84f8-415a-84f7-f8f7d9391a85-istiod-ca-cert\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.473937 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.473924 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-credential-socket\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.474043 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.474032 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-token\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.474165 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.474148 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-podinfo\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.475623 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.454046 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-data\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.475740 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.460424 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-workload-socket\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.475991 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.475960 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-workload-certs\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.489179 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.476271 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-credential-socket\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.489179 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.476825 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3acebec4-84f8-415a-84f7-f8f7d9391a85-istiod-ca-cert\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.505091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.497718 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-podinfo\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.505091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.499339 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-envoy\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.522783 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.522713 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-token\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.540098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.537122 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwkf4\" (UniqueName: \"kubernetes.io/projected/3acebec4-84f8-415a-84f7-f8f7d9391a85-kube-api-access-fwkf4\") pod \"e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.557779 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.557732 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:22.967688 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:27:22.967658 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3acebec4_84f8_415a_84f7_f8f7d9391a85.slice/crio-5d4e2c3162de31923a52447165bcc0966189ca9bbf2cfc9f14fc214b80c3bd06 WatchSource:0}: Error finding container 5d4e2c3162de31923a52447165bcc0966189ca9bbf2cfc9f14fc214b80c3bd06: Status 404 returned error can't find the container with id 5d4e2c3162de31923a52447165bcc0966189ca9bbf2cfc9f14fc214b80c3bd06 Oct 30 12:27:22.970397 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.970360 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:27:22.970510 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.970444 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:27:22.970577 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.970520 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:27:22.970697 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:22.970675 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r"] Oct 30 12:27:23.138334 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:23.138302 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" event={"ID":"3acebec4-84f8-415a-84f7-f8f7d9391a85","Type":"ContainerStarted","Data":"8013cb7d7a25a3866bb4699d4e861a05650bbb4e55c1fcffac5d0b78b20174e9"} Oct 30 12:27:23.138508 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:23.138369 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" event={"ID":"3acebec4-84f8-415a-84f7-f8f7d9391a85","Type":"ContainerStarted","Data":"5d4e2c3162de31923a52447165bcc0966189ca9bbf2cfc9f14fc214b80c3bd06"} Oct 30 12:27:23.211021 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:23.210963 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" podStartSLOduration=1.210946668 podStartE2EDuration="1.210946668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:27:23.190817949 +0000 UTC m=+1994.224956345" watchObservedRunningTime="2026-10-30 12:27:23.210946668 +0000 UTC m=+1994.245085064" Oct 30 12:27:23.558039 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:23.558007 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:23.602850 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:27:23.602806 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:27:24.597093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:24.596649 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:25.153438 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:25.153397 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:25.154740 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:25.154710 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:27:28.692188 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:28.688329 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd"] Oct 30 12:27:28.709248 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:28.709217 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd" Oct 30 12:27:28.721177 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:28.721139 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee03fe8b-948f-4a5d-90c0-14c97a9a71f5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd\" (UID: \"ee03fe8b-948f-4a5d-90c0-14c97a9a71f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd" Oct 30 12:27:28.721377 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:28.721253 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvkht\" (UniqueName: \"kubernetes.io/projected/ee03fe8b-948f-4a5d-90c0-14c97a9a71f5-kube-api-access-dvkht\") pod \"payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd\" (UID: \"ee03fe8b-948f-4a5d-90c0-14c97a9a71f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd" Oct 30 12:27:28.761353 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:28.761319 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7eee3b\"" Oct 30 12:27:28.765197 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:28.765167 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7eee3b-dockercfg-xltvg\"" Oct 30 12:27:28.778777 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:28.778737 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd"] Oct 30 12:27:28.822603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:28.822561 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee03fe8b-948f-4a5d-90c0-14c97a9a71f5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd\" (UID: \"ee03fe8b-948f-4a5d-90c0-14c97a9a71f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd" Oct 30 12:27:28.822811 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:28.822673 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvkht\" (UniqueName: \"kubernetes.io/projected/ee03fe8b-948f-4a5d-90c0-14c97a9a71f5-kube-api-access-dvkht\") pod \"payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd\" (UID: \"ee03fe8b-948f-4a5d-90c0-14c97a9a71f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd" Oct 30 12:27:28.823364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:28.823339 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee03fe8b-948f-4a5d-90c0-14c97a9a71f5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd\" (UID: \"ee03fe8b-948f-4a5d-90c0-14c97a9a71f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd" Oct 30 12:27:28.847593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:28.847563 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvkht\" (UniqueName: \"kubernetes.io/projected/ee03fe8b-948f-4a5d-90c0-14c97a9a71f5-kube-api-access-dvkht\") pod \"payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd\" (UID: \"ee03fe8b-948f-4a5d-90c0-14c97a9a71f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd" Oct 30 12:27:29.081993 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:29.081908 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd" Oct 30 12:27:29.104291 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:29.104262 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q" Oct 30 12:27:29.104694 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:29.104667 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt" Oct 30 12:27:29.423986 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:27:29.423957 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee03fe8b_948f_4a5d_90c0_14c97a9a71f5.slice/crio-362795c150d4a111ef1f55bdbdcd8b2265dbbd1fe5772db7c955dd8561f85124 WatchSource:0}: Error finding container 362795c150d4a111ef1f55bdbdcd8b2265dbbd1fe5772db7c955dd8561f85124: Status 404 returned error can't find the container with id 362795c150d4a111ef1f55bdbdcd8b2265dbbd1fe5772db7c955dd8561f85124 Oct 30 12:27:29.425696 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:29.425594 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd"] Oct 30 12:27:30.231721 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:30.231684 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd" event={"ID":"ee03fe8b-948f-4a5d-90c0-14c97a9a71f5","Type":"ContainerStarted","Data":"3feb0a78f49faf3317172fe4bb632b4b598e84224159abcdfa9a75867ac13f11"} Oct 30 12:27:30.232180 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:30.231742 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd" event={"ID":"ee03fe8b-948f-4a5d-90c0-14c97a9a71f5","Type":"ContainerStarted","Data":"362795c150d4a111ef1f55bdbdcd8b2265dbbd1fe5772db7c955dd8561f85124"} Oct 30 12:27:30.232180 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:30.231764 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd" Oct 30 12:27:30.292238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:30.292181 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd" podStartSLOduration=2.2921678070000002 podStartE2EDuration="2.292167807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:27:30.277352643 +0000 UTC m=+2001.311491038" watchObservedRunningTime="2026-10-30 12:27:30.292167807 +0000 UTC m=+2001.326306207" Oct 30 12:27:30.904523 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:30.904484 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8"] Oct 30 12:27:30.940009 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:30.939973 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl"] Oct 30 12:27:35.905337 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:35.905263 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8" podUID="28a586ed-6c97-48ed-b147-d197fa84ab91" containerName="payload-processing" containerID="cri-o://a401e7c18e8cf9b7e8442832b64ecd5d10df2907dd32b384be0221ff2435a52c" gracePeriod=25 Oct 30 12:27:35.940614 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:35.940574 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl" podUID="01d065fa-815a-4f94-8a46-17f9a3de8368" containerName="payload-pre-processing" containerID="cri-o://1be417ef86cfea9f4289300e6d762a3ef57240c776893e0e2d65d849aff69480" gracePeriod=25 Oct 30 12:27:36.266115 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.266084 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8" Oct 30 12:27:36.280481 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.280448 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a586ed-6c97-48ed-b147-d197fa84ab91-kube-api-access-mldmk\" (UniqueName: \"kubernetes.io/projected/28a586ed-6c97-48ed-b147-d197fa84ab91-kube-api-access-mldmk\") pod \"28a586ed-6c97-48ed-b147-d197fa84ab91\" (UID: \"28a586ed-6c97-48ed-b147-d197fa84ab91\") " Oct 30 12:27:36.280675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.280583 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28a586ed-6c97-48ed-b147-d197fa84ab91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a586ed-6c97-48ed-b147-d197fa84ab91-plugins-config-volume\") pod \"28a586ed-6c97-48ed-b147-d197fa84ab91\" (UID: \"28a586ed-6c97-48ed-b147-d197fa84ab91\") " Oct 30 12:27:36.281048 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.281022 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a586ed-6c97-48ed-b147-d197fa84ab91-plugins-config-volume" pod "28a586ed-6c97-48ed-b147-d197fa84ab91" (UID: "28a586ed-6c97-48ed-b147-d197fa84ab91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:27:36.281330 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.281305 4468 generic.go:356] "Generic (PLEG): container finished" podID="01d065fa-815a-4f94-8a46-17f9a3de8368" containerID="1be417ef86cfea9f4289300e6d762a3ef57240c776893e0e2d65d849aff69480" exitCode=0 Oct 30 12:27:36.281492 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.281470 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a586ed-6c97-48ed-b147-d197fa84ab91-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:27:36.281596 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.281487 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl" event={"ID":"01d065fa-815a-4f94-8a46-17f9a3de8368","Type":"ContainerDied","Data":"1be417ef86cfea9f4289300e6d762a3ef57240c776893e0e2d65d849aff69480"} Oct 30 12:27:36.283260 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.283235 4468 generic.go:356] "Generic (PLEG): container finished" podID="28a586ed-6c97-48ed-b147-d197fa84ab91" containerID="a401e7c18e8cf9b7e8442832b64ecd5d10df2907dd32b384be0221ff2435a52c" exitCode=0 Oct 30 12:27:36.283390 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.283371 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8" event={"ID":"28a586ed-6c97-48ed-b147-d197fa84ab91","Type":"ContainerDied","Data":"a401e7c18e8cf9b7e8442832b64ecd5d10df2907dd32b384be0221ff2435a52c"} Oct 30 12:27:36.283490 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.283422 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8" event={"ID":"28a586ed-6c97-48ed-b147-d197fa84ab91","Type":"ContainerDied","Data":"4154d78ebc5f97b2ca1de431ce275de11b38f61449a825ceb236c36cf4b0e56b"} Oct 30 12:27:36.283490 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.283414 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a586ed-6c97-48ed-b147-d197fa84ab91-kube-api-access-mldmk" pod "28a586ed-6c97-48ed-b147-d197fa84ab91" (UID: "28a586ed-6c97-48ed-b147-d197fa84ab91"). InnerVolumeSpecName "kube-api-access-mldmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:36.283490 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.283456 4468 scope.go:122] "RemoveContainer" containerID="a401e7c18e8cf9b7e8442832b64ecd5d10df2907dd32b384be0221ff2435a52c" Oct 30 12:27:36.283663 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.283527 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8" Oct 30 12:27:36.306501 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.306455 4468 scope.go:122] "RemoveContainer" containerID="a401e7c18e8cf9b7e8442832b64ecd5d10df2907dd32b384be0221ff2435a52c" Oct 30 12:27:36.307196 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:27:36.307159 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a401e7c18e8cf9b7e8442832b64ecd5d10df2907dd32b384be0221ff2435a52c\": container with ID starting with a401e7c18e8cf9b7e8442832b64ecd5d10df2907dd32b384be0221ff2435a52c not found: ID does not exist" containerID="a401e7c18e8cf9b7e8442832b64ecd5d10df2907dd32b384be0221ff2435a52c" Oct 30 12:27:36.307313 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.307207 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a401e7c18e8cf9b7e8442832b64ecd5d10df2907dd32b384be0221ff2435a52c"} err="failed to get container status \"a401e7c18e8cf9b7e8442832b64ecd5d10df2907dd32b384be0221ff2435a52c\": rpc error: code = NotFound desc = could not find container \"a401e7c18e8cf9b7e8442832b64ecd5d10df2907dd32b384be0221ff2435a52c\": container with ID starting with a401e7c18e8cf9b7e8442832b64ecd5d10df2907dd32b384be0221ff2435a52c not found: ID does not exist" Oct 30 12:27:36.311979 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.311954 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl" Oct 30 12:27:36.351743 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.351695 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8"] Oct 30 12:27:36.356784 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.356755 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-096834-68dccf59d-qk2x8"] Oct 30 12:27:36.382101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.382032 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01d065fa-815a-4f94-8a46-17f9a3de8368-kube-api-access-88q4m\" (UniqueName: \"kubernetes.io/projected/01d065fa-815a-4f94-8a46-17f9a3de8368-kube-api-access-88q4m\") pod \"01d065fa-815a-4f94-8a46-17f9a3de8368\" (UID: \"01d065fa-815a-4f94-8a46-17f9a3de8368\") " Oct 30 12:27:36.382288 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.382199 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01d065fa-815a-4f94-8a46-17f9a3de8368-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d065fa-815a-4f94-8a46-17f9a3de8368-plugins-config-volume\") pod \"01d065fa-815a-4f94-8a46-17f9a3de8368\" (UID: \"01d065fa-815a-4f94-8a46-17f9a3de8368\") " Oct 30 12:27:36.382709 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.382679 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d065fa-815a-4f94-8a46-17f9a3de8368-plugins-config-volume" pod "01d065fa-815a-4f94-8a46-17f9a3de8368" (UID: "01d065fa-815a-4f94-8a46-17f9a3de8368"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:27:36.383132 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.383109 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d065fa-815a-4f94-8a46-17f9a3de8368-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:27:36.383219 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.383139 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mldmk\" (UniqueName: \"kubernetes.io/projected/28a586ed-6c97-48ed-b147-d197fa84ab91-kube-api-access-mldmk\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:27:36.384397 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.384372 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d065fa-815a-4f94-8a46-17f9a3de8368-kube-api-access-88q4m" pod "01d065fa-815a-4f94-8a46-17f9a3de8368" (UID: "01d065fa-815a-4f94-8a46-17f9a3de8368"). InnerVolumeSpecName "kube-api-access-88q4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:36.484987 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.484887 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88q4m\" (UniqueName: \"kubernetes.io/projected/01d065fa-815a-4f94-8a46-17f9a3de8368-kube-api-access-88q4m\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:27:36.982496 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:36.982461 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554"] Oct 30 12:27:37.094996 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:37.094960 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a586ed-6c97-48ed-b147-d197fa84ab91" path="/var/lib/kubelet/pods/28a586ed-6c97-48ed-b147-d197fa84ab91/volumes" Oct 30 12:27:37.309199 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:37.309097 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl" event={"ID":"01d065fa-815a-4f94-8a46-17f9a3de8368","Type":"ContainerDied","Data":"fb0f9d431ef03ee3aa38f86695f7b02df5bb3410415bf9d10d4b549fcf676570"} Oct 30 12:27:37.309199 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:37.309180 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl" Oct 30 12:27:37.309199 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:37.309191 4468 scope.go:122] "RemoveContainer" containerID="1be417ef86cfea9f4289300e6d762a3ef57240c776893e0e2d65d849aff69480" Oct 30 12:27:37.423231 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:37.423198 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl"] Oct 30 12:27:37.432133 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:37.432092 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-096834-7cc5bd8f47-tmvjl"] Oct 30 12:27:39.096145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:39.096064 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01d065fa-815a-4f94-8a46-17f9a3de8368" path="/var/lib/kubelet/pods/01d065fa-815a-4f94-8a46-17f9a3de8368/volumes" Oct 30 12:27:41.242536 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:41.242505 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd" Oct 30 12:27:41.983341 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:41.983281 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554" podUID="c9f1a670-d9c8-4c44-bec0-10c84cf880f4" containerName="payload-pre-processing" containerID="cri-o://4db5015972d35d8334280b6a8efc264148cd1ea8fca166d0fee576a5686dc1e3" gracePeriod=25 Oct 30 12:27:42.274459 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:42.274427 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554" Oct 30 12:27:42.306999 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:42.306964 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f1a670-d9c8-4c44-bec0-10c84cf880f4-kube-api-access-pvnpd\" (UniqueName: \"kubernetes.io/projected/c9f1a670-d9c8-4c44-bec0-10c84cf880f4-kube-api-access-pvnpd\") pod \"c9f1a670-d9c8-4c44-bec0-10c84cf880f4\" (UID: \"c9f1a670-d9c8-4c44-bec0-10c84cf880f4\") " Oct 30 12:27:42.307225 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:42.307144 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9f1a670-d9c8-4c44-bec0-10c84cf880f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f1a670-d9c8-4c44-bec0-10c84cf880f4-plugins-config-volume\") pod \"c9f1a670-d9c8-4c44-bec0-10c84cf880f4\" (UID: \"c9f1a670-d9c8-4c44-bec0-10c84cf880f4\") " Oct 30 12:27:42.307522 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:42.307498 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f1a670-d9c8-4c44-bec0-10c84cf880f4-plugins-config-volume" pod "c9f1a670-d9c8-4c44-bec0-10c84cf880f4" (UID: "c9f1a670-d9c8-4c44-bec0-10c84cf880f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:27:42.307709 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:42.307696 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f1a670-d9c8-4c44-bec0-10c84cf880f4-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:27:42.309183 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:42.309161 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f1a670-d9c8-4c44-bec0-10c84cf880f4-kube-api-access-pvnpd" pod "c9f1a670-d9c8-4c44-bec0-10c84cf880f4" (UID: "c9f1a670-d9c8-4c44-bec0-10c84cf880f4"). InnerVolumeSpecName "kube-api-access-pvnpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:42.352734 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:42.352696 4468 generic.go:356] "Generic (PLEG): container finished" podID="c9f1a670-d9c8-4c44-bec0-10c84cf880f4" containerID="4db5015972d35d8334280b6a8efc264148cd1ea8fca166d0fee576a5686dc1e3" exitCode=0 Oct 30 12:27:42.352946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:42.352752 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554" Oct 30 12:27:42.352946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:42.352777 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554" event={"ID":"c9f1a670-d9c8-4c44-bec0-10c84cf880f4","Type":"ContainerDied","Data":"4db5015972d35d8334280b6a8efc264148cd1ea8fca166d0fee576a5686dc1e3"} Oct 30 12:27:42.352946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:42.352842 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554" event={"ID":"c9f1a670-d9c8-4c44-bec0-10c84cf880f4","Type":"ContainerDied","Data":"6c9bc27d28ceecf91d116125e7fca5ce3ad73d07af629f50754ace78e8917643"} Oct 30 12:27:42.352946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:42.352880 4468 scope.go:122] "RemoveContainer" containerID="4db5015972d35d8334280b6a8efc264148cd1ea8fca166d0fee576a5686dc1e3" Oct 30 12:27:42.365846 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:42.365821 4468 scope.go:122] "RemoveContainer" containerID="4db5015972d35d8334280b6a8efc264148cd1ea8fca166d0fee576a5686dc1e3" Oct 30 12:27:42.366261 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:27:42.366227 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db5015972d35d8334280b6a8efc264148cd1ea8fca166d0fee576a5686dc1e3\": container with ID starting with 4db5015972d35d8334280b6a8efc264148cd1ea8fca166d0fee576a5686dc1e3 not found: ID does not exist" containerID="4db5015972d35d8334280b6a8efc264148cd1ea8fca166d0fee576a5686dc1e3" Oct 30 12:27:42.366371 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:42.366270 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db5015972d35d8334280b6a8efc264148cd1ea8fca166d0fee576a5686dc1e3"} err="failed to get container status \"4db5015972d35d8334280b6a8efc264148cd1ea8fca166d0fee576a5686dc1e3\": rpc error: code = NotFound desc = could not find container \"4db5015972d35d8334280b6a8efc264148cd1ea8fca166d0fee576a5686dc1e3\": container with ID starting with 4db5015972d35d8334280b6a8efc264148cd1ea8fca166d0fee576a5686dc1e3 not found: ID does not exist" Oct 30 12:27:42.409078 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:42.409026 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvnpd\" (UniqueName: \"kubernetes.io/projected/c9f1a670-d9c8-4c44-bec0-10c84cf880f4-kube-api-access-pvnpd\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:27:42.412030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:42.411997 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554"] Oct 30 12:27:42.417153 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:42.417127 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c57af-5d768b897f-qs554"] Oct 30 12:27:43.090908 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:43.090870 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9f1a670-d9c8-4c44-bec0-10c84cf880f4" path="/var/lib/kubelet/pods/c9f1a670-d9c8-4c44-bec0-10c84cf880f4/volumes" Oct 30 12:27:53.815402 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:27:53.815313 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:27:58.075403 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:58.075338 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq"] Oct 30 12:27:58.075854 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:58.075633 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" podUID="4ad41b97-a567-4fc6-8d54-842e1547bfd7" containerName="istio-proxy" containerID="cri-o://b5bbd24cb0d88f0180ed98c526480cf8741bdd47be880e4a66db04f569ba4f7c" gracePeriod=30 Oct 30 12:27:58.591050 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:58.591016 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2"] Oct 30 12:27:58.591311 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:58.591262 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" podUID="90b0b79c-9489-4285-a629-aaf506cbecdc" containerName="maas-api" containerID="cri-o://90518be0484f68ecc5ef58610d7b9cd4c7d7bb77e2795e0fbdc78957defb1c00" gracePeriod=30 Oct 30 12:27:58.635844 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:58.635785 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" podUID="90b0b79c-9489-4285-a629-aaf506cbecdc" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.130:8443/health\": dial tcp 10.134.0.130:8443: connect: connection refused" Oct 30 12:27:58.768571 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:58.768535 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6"] Oct 30 12:27:58.768818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:58.768791 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6" podUID="b8d14547-06cf-4583-b119-c8fbd7aa1e95" containerName="payload-processing" containerID="cri-o://182a3c8e6e917ae071b136110031a7e0201c928b6ad902a46eecba71d579652c" gracePeriod=30 Oct 30 12:27:58.903935 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:58.903907 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" Oct 30 12:27:58.930268 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:58.929095 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90b0b79c-9489-4285-a629-aaf506cbecdc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90b0b79c-9489-4285-a629-aaf506cbecdc-maas-api-tls\") pod \"90b0b79c-9489-4285-a629-aaf506cbecdc\" (UID: \"90b0b79c-9489-4285-a629-aaf506cbecdc\") " Oct 30 12:27:58.930268 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:58.929199 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90b0b79c-9489-4285-a629-aaf506cbecdc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90b0b79c-9489-4285-a629-aaf506cbecdc-metrics-certs\") pod \"90b0b79c-9489-4285-a629-aaf506cbecdc\" (UID: \"90b0b79c-9489-4285-a629-aaf506cbecdc\") " Oct 30 12:27:58.930268 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:58.929257 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90b0b79c-9489-4285-a629-aaf506cbecdc-kube-api-access-2mhm7\" (UniqueName: \"kubernetes.io/projected/90b0b79c-9489-4285-a629-aaf506cbecdc-kube-api-access-2mhm7\") pod \"90b0b79c-9489-4285-a629-aaf506cbecdc\" (UID: \"90b0b79c-9489-4285-a629-aaf506cbecdc\") " Oct 30 12:27:58.930268 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:58.929297 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90b0b79c-9489-4285-a629-aaf506cbecdc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90b0b79c-9489-4285-a629-aaf506cbecdc-openshift-service-ca\") pod \"90b0b79c-9489-4285-a629-aaf506cbecdc\" (UID: \"90b0b79c-9489-4285-a629-aaf506cbecdc\") " Oct 30 12:27:58.930268 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:58.929930 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b0b79c-9489-4285-a629-aaf506cbecdc-openshift-service-ca" pod "90b0b79c-9489-4285-a629-aaf506cbecdc" (UID: "90b0b79c-9489-4285-a629-aaf506cbecdc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:27:58.934832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:58.934139 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90b0b79c-9489-4285-a629-aaf506cbecdc-maas-api-tls" pod "90b0b79c-9489-4285-a629-aaf506cbecdc" (UID: "90b0b79c-9489-4285-a629-aaf506cbecdc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:27:58.934832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:58.934153 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b0b79c-9489-4285-a629-aaf506cbecdc-kube-api-access-2mhm7" pod "90b0b79c-9489-4285-a629-aaf506cbecdc" (UID: "90b0b79c-9489-4285-a629-aaf506cbecdc"). InnerVolumeSpecName "kube-api-access-2mhm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:58.941571 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:58.941536 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90b0b79c-9489-4285-a629-aaf506cbecdc-metrics-certs" pod "90b0b79c-9489-4285-a629-aaf506cbecdc" (UID: "90b0b79c-9489-4285-a629-aaf506cbecdc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:27:59.031325 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.031274 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90b0b79c-9489-4285-a629-aaf506cbecdc-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:27:59.031325 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.031315 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mhm7\" (UniqueName: \"kubernetes.io/projected/90b0b79c-9489-4285-a629-aaf506cbecdc-kube-api-access-2mhm7\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:27:59.031325 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.031330 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90b0b79c-9489-4285-a629-aaf506cbecdc-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:27:59.031676 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.031345 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90b0b79c-9489-4285-a629-aaf506cbecdc-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:27:59.054631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.054599 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6" Oct 30 12:27:59.146142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.140516 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d14547-06cf-4583-b119-c8fbd7aa1e95-kube-api-access-7p2lj\" (UniqueName: \"kubernetes.io/projected/b8d14547-06cf-4583-b119-c8fbd7aa1e95-kube-api-access-7p2lj\") pod \"b8d14547-06cf-4583-b119-c8fbd7aa1e95\" (UID: \"b8d14547-06cf-4583-b119-c8fbd7aa1e95\") " Oct 30 12:27:59.146142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.140713 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d14547-06cf-4583-b119-c8fbd7aa1e95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d14547-06cf-4583-b119-c8fbd7aa1e95-plugins-config-volume\") pod \"b8d14547-06cf-4583-b119-c8fbd7aa1e95\" (UID: \"b8d14547-06cf-4583-b119-c8fbd7aa1e95\") " Oct 30 12:27:59.146142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.143115 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d14547-06cf-4583-b119-c8fbd7aa1e95-plugins-config-volume" pod "b8d14547-06cf-4583-b119-c8fbd7aa1e95" (UID: "b8d14547-06cf-4583-b119-c8fbd7aa1e95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:27:59.159093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.157789 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d14547-06cf-4583-b119-c8fbd7aa1e95-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:27:59.255091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.252386 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d14547-06cf-4583-b119-c8fbd7aa1e95-kube-api-access-7p2lj" pod "b8d14547-06cf-4583-b119-c8fbd7aa1e95" (UID: "b8d14547-06cf-4583-b119-c8fbd7aa1e95"). InnerVolumeSpecName "kube-api-access-7p2lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:59.267090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.264265 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p2lj\" (UniqueName: \"kubernetes.io/projected/b8d14547-06cf-4583-b119-c8fbd7aa1e95-kube-api-access-7p2lj\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:27:59.709586 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.709553 4468 generic.go:356] "Generic (PLEG): container finished" podID="b8d14547-06cf-4583-b119-c8fbd7aa1e95" containerID="182a3c8e6e917ae071b136110031a7e0201c928b6ad902a46eecba71d579652c" exitCode=0 Oct 30 12:27:59.709753 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.709606 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6" Oct 30 12:27:59.709753 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.709631 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6" event={"ID":"b8d14547-06cf-4583-b119-c8fbd7aa1e95","Type":"ContainerDied","Data":"182a3c8e6e917ae071b136110031a7e0201c928b6ad902a46eecba71d579652c"} Oct 30 12:27:59.709753 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.709685 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6" event={"ID":"b8d14547-06cf-4583-b119-c8fbd7aa1e95","Type":"ContainerDied","Data":"1d9f6e5a01016b7930461bbd136ec571bf931fc6483aa4a6dc907303f2edecd5"} Oct 30 12:27:59.709753 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.709716 4468 scope.go:122] "RemoveContainer" containerID="182a3c8e6e917ae071b136110031a7e0201c928b6ad902a46eecba71d579652c" Oct 30 12:27:59.711266 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.711246 4468 generic.go:356] "Generic (PLEG): container finished" podID="90b0b79c-9489-4285-a629-aaf506cbecdc" containerID="90518be0484f68ecc5ef58610d7b9cd4c7d7bb77e2795e0fbdc78957defb1c00" exitCode=0 Oct 30 12:27:59.711363 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.711290 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" Oct 30 12:27:59.711363 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.711322 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" event={"ID":"90b0b79c-9489-4285-a629-aaf506cbecdc","Type":"ContainerDied","Data":"90518be0484f68ecc5ef58610d7b9cd4c7d7bb77e2795e0fbdc78957defb1c00"} Oct 30 12:27:59.711472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.711365 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2" event={"ID":"90b0b79c-9489-4285-a629-aaf506cbecdc","Type":"ContainerDied","Data":"6afb4834e8fb2f273804bba29e2bee7fee7c08f7009c5aded8f3788893fa2da4"} Oct 30 12:27:59.721147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.721131 4468 scope.go:122] "RemoveContainer" containerID="182a3c8e6e917ae071b136110031a7e0201c928b6ad902a46eecba71d579652c" Oct 30 12:27:59.721440 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:27:59.721408 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"182a3c8e6e917ae071b136110031a7e0201c928b6ad902a46eecba71d579652c\": container with ID starting with 182a3c8e6e917ae071b136110031a7e0201c928b6ad902a46eecba71d579652c not found: ID does not exist" containerID="182a3c8e6e917ae071b136110031a7e0201c928b6ad902a46eecba71d579652c" Oct 30 12:27:59.721535 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.721447 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"182a3c8e6e917ae071b136110031a7e0201c928b6ad902a46eecba71d579652c"} err="failed to get container status \"182a3c8e6e917ae071b136110031a7e0201c928b6ad902a46eecba71d579652c\": rpc error: code = NotFound desc = could not find container \"182a3c8e6e917ae071b136110031a7e0201c928b6ad902a46eecba71d579652c\": container with ID starting with 182a3c8e6e917ae071b136110031a7e0201c928b6ad902a46eecba71d579652c not found: ID does not exist" Oct 30 12:27:59.721535 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.721474 4468 scope.go:122] "RemoveContainer" containerID="90518be0484f68ecc5ef58610d7b9cd4c7d7bb77e2795e0fbdc78957defb1c00" Oct 30 12:27:59.732182 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.732160 4468 scope.go:122] "RemoveContainer" containerID="90518be0484f68ecc5ef58610d7b9cd4c7d7bb77e2795e0fbdc78957defb1c00" Oct 30 12:27:59.732488 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:27:59.732464 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90518be0484f68ecc5ef58610d7b9cd4c7d7bb77e2795e0fbdc78957defb1c00\": container with ID starting with 90518be0484f68ecc5ef58610d7b9cd4c7d7bb77e2795e0fbdc78957defb1c00 not found: ID does not exist" containerID="90518be0484f68ecc5ef58610d7b9cd4c7d7bb77e2795e0fbdc78957defb1c00" Oct 30 12:27:59.732583 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.732497 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90518be0484f68ecc5ef58610d7b9cd4c7d7bb77e2795e0fbdc78957defb1c00"} err="failed to get container status \"90518be0484f68ecc5ef58610d7b9cd4c7d7bb77e2795e0fbdc78957defb1c00\": rpc error: code = NotFound desc = could not find container \"90518be0484f68ecc5ef58610d7b9cd4c7d7bb77e2795e0fbdc78957defb1c00\": container with ID starting with 90518be0484f68ecc5ef58610d7b9cd4c7d7bb77e2795e0fbdc78957defb1c00 not found: ID does not exist" Oct 30 12:27:59.768967 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.768935 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6"] Oct 30 12:27:59.782052 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.782023 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e2db2-65bd664f7d-h27f6"] Oct 30 12:27:59.812776 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.812726 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2"] Oct 30 12:27:59.817743 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:27:59.817713 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e2db2-666dfc658c-dcnf2"] Oct 30 12:28:01.090969 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:01.090934 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90b0b79c-9489-4285-a629-aaf506cbecdc" path="/var/lib/kubelet/pods/90b0b79c-9489-4285-a629-aaf506cbecdc/volumes" Oct 30 12:28:01.091509 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:01.091487 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d14547-06cf-4583-b119-c8fbd7aa1e95" path="/var/lib/kubelet/pods/b8d14547-06cf-4583-b119-c8fbd7aa1e95/volumes" Oct 30 12:28:03.303906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.301204 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22"] Oct 30 12:28:03.303906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.302740 4468 cpu_manager.go:414] "containerMap: removing container" podUID="c9f1a670-d9c8-4c44-bec0-10c84cf880f4" containerName="payload-pre-processing" Oct 30 12:28:03.303906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.302767 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f1a670-d9c8-4c44-bec0-10c84cf880f4" containerName="payload-pre-processing" Oct 30 12:28:03.303906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.302804 4468 cpu_manager.go:414] "containerMap: removing container" podUID="b8d14547-06cf-4583-b119-c8fbd7aa1e95" containerName="payload-processing" Oct 30 12:28:03.303906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.302813 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d14547-06cf-4583-b119-c8fbd7aa1e95" containerName="payload-processing" Oct 30 12:28:03.303906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.302829 4468 cpu_manager.go:414] "containerMap: removing container" podUID="01d065fa-815a-4f94-8a46-17f9a3de8368" containerName="payload-pre-processing" Oct 30 12:28:03.303906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.302838 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d065fa-815a-4f94-8a46-17f9a3de8368" containerName="payload-pre-processing" Oct 30 12:28:03.303906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.302895 4468 cpu_manager.go:414] "containerMap: removing container" podUID="90b0b79c-9489-4285-a629-aaf506cbecdc" containerName="maas-api" Oct 30 12:28:03.303906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.302903 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90b0b79c-9489-4285-a629-aaf506cbecdc" containerName="maas-api" Oct 30 12:28:03.303906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.302916 4468 cpu_manager.go:414] "containerMap: removing container" podUID="28a586ed-6c97-48ed-b147-d197fa84ab91" containerName="payload-processing" Oct 30 12:28:03.303906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.302924 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a586ed-6c97-48ed-b147-d197fa84ab91" containerName="payload-processing" Oct 30 12:28:03.303906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.303112 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="90b0b79c-9489-4285-a629-aaf506cbecdc" containerName="maas-api" Oct 30 12:28:03.303906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.303126 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a586ed-6c97-48ed-b147-d197fa84ab91" containerName="payload-processing" Oct 30 12:28:03.303906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.303142 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f1a670-d9c8-4c44-bec0-10c84cf880f4" containerName="payload-pre-processing" Oct 30 12:28:03.303906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.303158 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d14547-06cf-4583-b119-c8fbd7aa1e95" containerName="payload-processing" Oct 30 12:28:03.303906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.303167 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d065fa-815a-4f94-8a46-17f9a3de8368" containerName="payload-pre-processing" Oct 30 12:28:03.309878 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.308468 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22"] Oct 30 12:28:03.309878 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.308582 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22" Oct 30 12:28:03.338984 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.338952 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5c57af-dockercfg-f4q6b\"" Oct 30 12:28:03.339214 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.338964 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5c57af\"" Oct 30 12:28:03.491810 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.491771 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d6w6\" (UniqueName: \"kubernetes.io/projected/b6055421-f8de-4e98-bd0e-4f1a0cbeefc5-kube-api-access-6d6w6\") pod \"payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22\" (UID: \"b6055421-f8de-4e98-bd0e-4f1a0cbeefc5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22" Oct 30 12:28:03.492037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.491972 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6055421-f8de-4e98-bd0e-4f1a0cbeefc5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22\" (UID: \"b6055421-f8de-4e98-bd0e-4f1a0cbeefc5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22" Oct 30 12:28:03.593819 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.593781 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d6w6\" (UniqueName: \"kubernetes.io/projected/b6055421-f8de-4e98-bd0e-4f1a0cbeefc5-kube-api-access-6d6w6\") pod \"payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22\" (UID: \"b6055421-f8de-4e98-bd0e-4f1a0cbeefc5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22" Oct 30 12:28:03.593994 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.593880 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6055421-f8de-4e98-bd0e-4f1a0cbeefc5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22\" (UID: \"b6055421-f8de-4e98-bd0e-4f1a0cbeefc5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22" Oct 30 12:28:03.594667 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.594640 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6055421-f8de-4e98-bd0e-4f1a0cbeefc5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22\" (UID: \"b6055421-f8de-4e98-bd0e-4f1a0cbeefc5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22" Oct 30 12:28:03.619146 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.619115 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d6w6\" (UniqueName: \"kubernetes.io/projected/b6055421-f8de-4e98-bd0e-4f1a0cbeefc5-kube-api-access-6d6w6\") pod \"payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22\" (UID: \"b6055421-f8de-4e98-bd0e-4f1a0cbeefc5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22" Oct 30 12:28:03.629916 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.629887 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22" Oct 30 12:28:03.790803 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.790766 4468 generic.go:356] "Generic (PLEG): container finished" podID="4ad41b97-a567-4fc6-8d54-842e1547bfd7" containerID="b5bbd24cb0d88f0180ed98c526480cf8741bdd47be880e4a66db04f569ba4f7c" exitCode=0 Oct 30 12:28:03.790965 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.790837 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" event={"ID":"4ad41b97-a567-4fc6-8d54-842e1547bfd7","Type":"ContainerDied","Data":"b5bbd24cb0d88f0180ed98c526480cf8741bdd47be880e4a66db04f569ba4f7c"} Oct 30 12:28:03.864057 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.864033 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:28:03.999664 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:03.999636 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22"] Oct 30 12:28:04.005055 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.005029 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-workload-certs\") pod \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " Oct 30 12:28:04.005238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.005218 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad41b97-a567-4fc6-8d54-842e1547bfd7-kube-api-access-9txl6\" (UniqueName: \"kubernetes.io/projected/4ad41b97-a567-4fc6-8d54-842e1547bfd7-kube-api-access-9txl6\") pod \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " Oct 30 12:28:04.005318 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.005298 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-envoy\") pod \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " Oct 30 12:28:04.005473 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.005335 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-token\" (UniqueName: \"kubernetes.io/projected/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-token\") pod \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " Oct 30 12:28:04.005473 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.005348 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-workload-certs" pod "4ad41b97-a567-4fc6-8d54-842e1547bfd7" (UID: "4ad41b97-a567-4fc6-8d54-842e1547bfd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:04.005473 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.005436 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-data\") pod \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " Oct 30 12:28:04.005596 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.005482 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istiod-ca-cert\") pod \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " Oct 30 12:28:04.005596 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.005529 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-workload-socket\") pod \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " Oct 30 12:28:04.005596 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.005586 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-credential-socket\") pod \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " Oct 30 12:28:04.005766 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.005620 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-podinfo\") pod \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\" (UID: \"4ad41b97-a567-4fc6-8d54-842e1547bfd7\") " Oct 30 12:28:04.005846 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.005802 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-data" pod "4ad41b97-a567-4fc6-8d54-842e1547bfd7" (UID: "4ad41b97-a567-4fc6-8d54-842e1547bfd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:04.005912 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.005853 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-workload-socket" pod "4ad41b97-a567-4fc6-8d54-842e1547bfd7" (UID: "4ad41b97-a567-4fc6-8d54-842e1547bfd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:04.005966 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.005901 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-credential-socket" pod "4ad41b97-a567-4fc6-8d54-842e1547bfd7" (UID: "4ad41b97-a567-4fc6-8d54-842e1547bfd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:04.006255 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.006230 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istiod-ca-cert" pod "4ad41b97-a567-4fc6-8d54-842e1547bfd7" (UID: "4ad41b97-a567-4fc6-8d54-842e1547bfd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:04.006516 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.006492 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:04.006593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.006522 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:04.006593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.006538 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:04.006593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.006550 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:04.006593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.006563 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:04.007907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.007876 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-podinfo" pod "4ad41b97-a567-4fc6-8d54-842e1547bfd7" (UID: "4ad41b97-a567-4fc6-8d54-842e1547bfd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:28:04.007979 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.007914 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-token" pod "4ad41b97-a567-4fc6-8d54-842e1547bfd7" (UID: "4ad41b97-a567-4fc6-8d54-842e1547bfd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:04.008288 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.008265 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-envoy" pod "4ad41b97-a567-4fc6-8d54-842e1547bfd7" (UID: "4ad41b97-a567-4fc6-8d54-842e1547bfd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:04.008496 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.008477 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad41b97-a567-4fc6-8d54-842e1547bfd7-kube-api-access-9txl6" pod "4ad41b97-a567-4fc6-8d54-842e1547bfd7" (UID: "4ad41b97-a567-4fc6-8d54-842e1547bfd7"). InnerVolumeSpecName "kube-api-access-9txl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:04.108004 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.107969 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9txl6\" (UniqueName: \"kubernetes.io/projected/4ad41b97-a567-4fc6-8d54-842e1547bfd7-kube-api-access-9txl6\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:04.108004 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.108003 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:04.108266 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.108018 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:04.108266 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.108029 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad41b97-a567-4fc6-8d54-842e1547bfd7-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:04.816103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.811504 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" event={"ID":"4ad41b97-a567-4fc6-8d54-842e1547bfd7","Type":"ContainerDied","Data":"cf5ce32df7fbce4a1def2ee207c9f96d145aa95ccf3b8429e8109121cc0fedf1"} Oct 30 12:28:04.816103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.811582 4468 scope.go:122] "RemoveContainer" containerID="b5bbd24cb0d88f0180ed98c526480cf8741bdd47be880e4a66db04f569ba4f7c" Oct 30 12:28:04.816103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.811735 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq" Oct 30 12:28:04.824442 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.824397 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22" event={"ID":"b6055421-f8de-4e98-bd0e-4f1a0cbeefc5","Type":"ContainerStarted","Data":"5d0b9aa6e6b04c55b7144384b93da04556af9f01e31879f7110326a440d80dca"} Oct 30 12:28:04.824600 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.824491 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22" event={"ID":"b6055421-f8de-4e98-bd0e-4f1a0cbeefc5","Type":"ContainerStarted","Data":"ddbd883eca4ba1b25c44bebcd4d3693d380e36b36bf70fcd90b491dddcb844a4"} Oct 30 12:28:04.825199 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.825176 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22" Oct 30 12:28:04.885380 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.885325 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22" podStartSLOduration=1.885310992 podStartE2EDuration="1.885310992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:04.88276803 +0000 UTC m=+2035.916906426" watchObservedRunningTime="2026-10-30 12:28:04.885310992 +0000 UTC m=+2035.919449388" Oct 30 12:28:04.927489 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.927453 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq"] Oct 30 12:28:04.936643 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:04.936617 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpkqq"] Oct 30 12:28:05.090929 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:05.090843 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad41b97-a567-4fc6-8d54-842e1547bfd7" path="/var/lib/kubelet/pods/4ad41b97-a567-4fc6-8d54-842e1547bfd7/volumes" Oct 30 12:28:06.886751 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:06.886713 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2"] Oct 30 12:28:06.887244 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:06.886981 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" podUID="efb7c9be-be70-4b37-8657-95c9a1e283ca" containerName="istio-proxy" containerID="cri-o://c599648ca47ec8982574e7c49174958e8d2cf23ca96ba7cc9cc6f51c8b3945ad" gracePeriod=30 Oct 30 12:28:12.205097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.205054 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:28:12.259542 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.259510 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-envoy\") pod \"efb7c9be-be70-4b37-8657-95c9a1e283ca\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " Oct 30 12:28:12.259705 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.259602 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efb7c9be-be70-4b37-8657-95c9a1e283ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb7c9be-be70-4b37-8657-95c9a1e283ca-istiod-ca-cert\") pod \"efb7c9be-be70-4b37-8657-95c9a1e283ca\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " Oct 30 12:28:12.259705 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.259652 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-workload-certs\") pod \"efb7c9be-be70-4b37-8657-95c9a1e283ca\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " Oct 30 12:28:12.259817 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.259754 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-token\" (UniqueName: \"kubernetes.io/projected/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-token\") pod \"efb7c9be-be70-4b37-8657-95c9a1e283ca\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " Oct 30 12:28:12.259876 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.259824 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb7c9be-be70-4b37-8657-95c9a1e283ca-kube-api-access-8lhf9\" (UniqueName: \"kubernetes.io/projected/efb7c9be-be70-4b37-8657-95c9a1e283ca-kube-api-access-8lhf9\") pod \"efb7c9be-be70-4b37-8657-95c9a1e283ca\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " Oct 30 12:28:12.259934 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.259881 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-data\") pod \"efb7c9be-be70-4b37-8657-95c9a1e283ca\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " Oct 30 12:28:12.259934 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.259919 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-podinfo\") pod \"efb7c9be-be70-4b37-8657-95c9a1e283ca\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " Oct 30 12:28:12.260034 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.259969 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-workload-socket\") pod \"efb7c9be-be70-4b37-8657-95c9a1e283ca\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " Oct 30 12:28:12.260034 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.260003 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-workload-certs" pod "efb7c9be-be70-4b37-8657-95c9a1e283ca" (UID: "efb7c9be-be70-4b37-8657-95c9a1e283ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:12.260166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.260033 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb7c9be-be70-4b37-8657-95c9a1e283ca-istiod-ca-cert" pod "efb7c9be-be70-4b37-8657-95c9a1e283ca" (UID: "efb7c9be-be70-4b37-8657-95c9a1e283ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:12.260166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.260050 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-credential-socket\") pod \"efb7c9be-be70-4b37-8657-95c9a1e283ca\" (UID: \"efb7c9be-be70-4b37-8657-95c9a1e283ca\") " Oct 30 12:28:12.260264 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.260233 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-data" pod "efb7c9be-be70-4b37-8657-95c9a1e283ca" (UID: "efb7c9be-be70-4b37-8657-95c9a1e283ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:12.260487 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.260418 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-workload-socket" pod "efb7c9be-be70-4b37-8657-95c9a1e283ca" (UID: "efb7c9be-be70-4b37-8657-95c9a1e283ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:12.260487 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.260450 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-credential-socket" pod "efb7c9be-be70-4b37-8657-95c9a1e283ca" (UID: "efb7c9be-be70-4b37-8657-95c9a1e283ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:12.260909 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.260885 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:12.260983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.260913 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:12.260983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.260926 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:12.260983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.260940 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb7c9be-be70-4b37-8657-95c9a1e283ca-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:12.260983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.260954 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:12.261823 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.261801 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-envoy" pod "efb7c9be-be70-4b37-8657-95c9a1e283ca" (UID: "efb7c9be-be70-4b37-8657-95c9a1e283ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:12.262063 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.262039 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-token" pod "efb7c9be-be70-4b37-8657-95c9a1e283ca" (UID: "efb7c9be-be70-4b37-8657-95c9a1e283ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:12.262063 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.262081 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb7c9be-be70-4b37-8657-95c9a1e283ca-kube-api-access-8lhf9" pod "efb7c9be-be70-4b37-8657-95c9a1e283ca" (UID: "efb7c9be-be70-4b37-8657-95c9a1e283ca"). InnerVolumeSpecName "kube-api-access-8lhf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:12.262239 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.262170 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-podinfo" pod "efb7c9be-be70-4b37-8657-95c9a1e283ca" (UID: "efb7c9be-be70-4b37-8657-95c9a1e283ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:28:12.362338 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.362303 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:12.362338 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.362331 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:12.362338 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.362341 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lhf9\" (UniqueName: \"kubernetes.io/projected/efb7c9be-be70-4b37-8657-95c9a1e283ca-kube-api-access-8lhf9\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:12.362551 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.362351 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb7c9be-be70-4b37-8657-95c9a1e283ca-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:12.885864 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.885827 4468 generic.go:356] "Generic (PLEG): container finished" podID="efb7c9be-be70-4b37-8657-95c9a1e283ca" containerID="c599648ca47ec8982574e7c49174958e8d2cf23ca96ba7cc9cc6f51c8b3945ad" exitCode=0 Oct 30 12:28:12.886035 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.885886 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" Oct 30 12:28:12.886035 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.885915 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" event={"ID":"efb7c9be-be70-4b37-8657-95c9a1e283ca","Type":"ContainerDied","Data":"c599648ca47ec8982574e7c49174958e8d2cf23ca96ba7cc9cc6f51c8b3945ad"} Oct 30 12:28:12.886035 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.885972 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2" event={"ID":"efb7c9be-be70-4b37-8657-95c9a1e283ca","Type":"ContainerDied","Data":"ad9ba23ae882456f5c28f921b35dc59f394ebcc0b2242c3a6a9b9c54f2348446"} Oct 30 12:28:12.886035 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.885995 4468 scope.go:122] "RemoveContainer" containerID="c599648ca47ec8982574e7c49174958e8d2cf23ca96ba7cc9cc6f51c8b3945ad" Oct 30 12:28:12.897092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.897047 4468 scope.go:122] "RemoveContainer" containerID="49918c97bcec47db9860abf33290cf90c23f9f1de00b1c6c8453578a4c4bbee0" Oct 30 12:28:12.906057 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.906039 4468 scope.go:122] "RemoveContainer" containerID="c599648ca47ec8982574e7c49174958e8d2cf23ca96ba7cc9cc6f51c8b3945ad" Oct 30 12:28:12.906373 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:28:12.906349 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c599648ca47ec8982574e7c49174958e8d2cf23ca96ba7cc9cc6f51c8b3945ad\": container with ID starting with c599648ca47ec8982574e7c49174958e8d2cf23ca96ba7cc9cc6f51c8b3945ad not found: ID does not exist" containerID="c599648ca47ec8982574e7c49174958e8d2cf23ca96ba7cc9cc6f51c8b3945ad" Oct 30 12:28:12.906443 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.906380 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c599648ca47ec8982574e7c49174958e8d2cf23ca96ba7cc9cc6f51c8b3945ad"} err="failed to get container status \"c599648ca47ec8982574e7c49174958e8d2cf23ca96ba7cc9cc6f51c8b3945ad\": rpc error: code = NotFound desc = could not find container \"c599648ca47ec8982574e7c49174958e8d2cf23ca96ba7cc9cc6f51c8b3945ad\": container with ID starting with c599648ca47ec8982574e7c49174958e8d2cf23ca96ba7cc9cc6f51c8b3945ad not found: ID does not exist" Oct 30 12:28:12.906443 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.906398 4468 scope.go:122] "RemoveContainer" containerID="49918c97bcec47db9860abf33290cf90c23f9f1de00b1c6c8453578a4c4bbee0" Oct 30 12:28:12.906685 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:28:12.906666 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49918c97bcec47db9860abf33290cf90c23f9f1de00b1c6c8453578a4c4bbee0\": container with ID starting with 49918c97bcec47db9860abf33290cf90c23f9f1de00b1c6c8453578a4c4bbee0 not found: ID does not exist" containerID="49918c97bcec47db9860abf33290cf90c23f9f1de00b1c6c8453578a4c4bbee0" Oct 30 12:28:12.906730 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.906691 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49918c97bcec47db9860abf33290cf90c23f9f1de00b1c6c8453578a4c4bbee0"} err="failed to get container status \"49918c97bcec47db9860abf33290cf90c23f9f1de00b1c6c8453578a4c4bbee0\": rpc error: code = NotFound desc = could not find container \"49918c97bcec47db9860abf33290cf90c23f9f1de00b1c6c8453578a4c4bbee0\": container with ID starting with 49918c97bcec47db9860abf33290cf90c23f9f1de00b1c6c8453578a4c4bbee0 not found: ID does not exist" Oct 30 12:28:12.961976 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.961946 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2"] Oct 30 12:28:12.975469 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:12.975440 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7e2db2-openshift-default-547d4f89dc-w8xd2"] Oct 30 12:28:13.090941 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:13.090913 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efb7c9be-be70-4b37-8657-95c9a1e283ca" path="/var/lib/kubelet/pods/efb7c9be-be70-4b37-8657-95c9a1e283ca/volumes" Oct 30 12:28:13.566585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:13.566550 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q"] Oct 30 12:28:13.567060 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:13.566787 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q" podUID="5503f728-1761-4eba-905c-47d8e4bb6677" containerName="payload-processing" containerID="cri-o://35cf38cdaf3749007196d2100518aed9251b2456e218446df0ae504149178116" gracePeriod=30 Oct 30 12:28:13.578100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:13.578051 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt"] Oct 30 12:28:13.578331 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:13.578295 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt" podUID="fafb0ac3-55ef-4d91-ba47-4ad9b392b988" containerName="payload-pre-processing" containerID="cri-o://3624bcf3b0467397cb6e0d85953a203fbb5548c9e7cd392dd026e59ac11e2ca9" gracePeriod=30 Oct 30 12:28:14.011135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:14.010953 4468 generic.go:356] "Generic (PLEG): container finished" podID="5503f728-1761-4eba-905c-47d8e4bb6677" containerID="35cf38cdaf3749007196d2100518aed9251b2456e218446df0ae504149178116" exitCode=0 Oct 30 12:28:14.011135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:14.011047 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q" event={"ID":"5503f728-1761-4eba-905c-47d8e4bb6677","Type":"ContainerDied","Data":"35cf38cdaf3749007196d2100518aed9251b2456e218446df0ae504149178116"} Oct 30 12:28:14.023574 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:14.023268 4468 generic.go:356] "Generic (PLEG): container finished" podID="fafb0ac3-55ef-4d91-ba47-4ad9b392b988" containerID="3624bcf3b0467397cb6e0d85953a203fbb5548c9e7cd392dd026e59ac11e2ca9" exitCode=0 Oct 30 12:28:14.023574 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:14.023497 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt" event={"ID":"fafb0ac3-55ef-4d91-ba47-4ad9b392b988","Type":"ContainerDied","Data":"3624bcf3b0467397cb6e0d85953a203fbb5548c9e7cd392dd026e59ac11e2ca9"} Oct 30 12:28:14.055181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:14.050856 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q" Oct 30 12:28:14.062587 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:14.062563 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt" Oct 30 12:28:14.119480 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:14.119445 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fafb0ac3-55ef-4d91-ba47-4ad9b392b988-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fafb0ac3-55ef-4d91-ba47-4ad9b392b988-plugins-config-volume\") pod \"fafb0ac3-55ef-4d91-ba47-4ad9b392b988\" (UID: \"fafb0ac3-55ef-4d91-ba47-4ad9b392b988\") " Oct 30 12:28:14.119683 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:14.119598 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fafb0ac3-55ef-4d91-ba47-4ad9b392b988-kube-api-access-k8zxc\" (UniqueName: \"kubernetes.io/projected/fafb0ac3-55ef-4d91-ba47-4ad9b392b988-kube-api-access-k8zxc\") pod \"fafb0ac3-55ef-4d91-ba47-4ad9b392b988\" (UID: \"fafb0ac3-55ef-4d91-ba47-4ad9b392b988\") " Oct 30 12:28:14.119765 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:14.119740 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5503f728-1761-4eba-905c-47d8e4bb6677-kube-api-access-zbm4l\" (UniqueName: \"kubernetes.io/projected/5503f728-1761-4eba-905c-47d8e4bb6677-kube-api-access-zbm4l\") pod \"5503f728-1761-4eba-905c-47d8e4bb6677\" (UID: \"5503f728-1761-4eba-905c-47d8e4bb6677\") " Oct 30 12:28:14.119827 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:14.119791 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5503f728-1761-4eba-905c-47d8e4bb6677-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5503f728-1761-4eba-905c-47d8e4bb6677-plugins-config-volume\") pod \"5503f728-1761-4eba-905c-47d8e4bb6677\" (UID: \"5503f728-1761-4eba-905c-47d8e4bb6677\") " Oct 30 12:28:14.119881 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:14.119856 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fafb0ac3-55ef-4d91-ba47-4ad9b392b988-plugins-config-volume" pod "fafb0ac3-55ef-4d91-ba47-4ad9b392b988" (UID: "fafb0ac3-55ef-4d91-ba47-4ad9b392b988"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:14.120242 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:14.120208 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5503f728-1761-4eba-905c-47d8e4bb6677-plugins-config-volume" pod "5503f728-1761-4eba-905c-47d8e4bb6677" (UID: "5503f728-1761-4eba-905c-47d8e4bb6677"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:14.120516 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:14.120497 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fafb0ac3-55ef-4d91-ba47-4ad9b392b988-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:14.120633 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:14.120521 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5503f728-1761-4eba-905c-47d8e4bb6677-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:14.122386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:14.122357 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fafb0ac3-55ef-4d91-ba47-4ad9b392b988-kube-api-access-k8zxc" pod "fafb0ac3-55ef-4d91-ba47-4ad9b392b988" (UID: "fafb0ac3-55ef-4d91-ba47-4ad9b392b988"). InnerVolumeSpecName "kube-api-access-k8zxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:14.122502 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:14.122477 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5503f728-1761-4eba-905c-47d8e4bb6677-kube-api-access-zbm4l" pod "5503f728-1761-4eba-905c-47d8e4bb6677" (UID: "5503f728-1761-4eba-905c-47d8e4bb6677"). InnerVolumeSpecName "kube-api-access-zbm4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:14.222491 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:14.222451 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8zxc\" (UniqueName: \"kubernetes.io/projected/fafb0ac3-55ef-4d91-ba47-4ad9b392b988-kube-api-access-k8zxc\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:14.222491 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:14.222482 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbm4l\" (UniqueName: \"kubernetes.io/projected/5503f728-1761-4eba-905c-47d8e4bb6677-kube-api-access-zbm4l\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:15.056802 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:15.056774 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q" Oct 30 12:28:15.057334 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:15.056774 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q" event={"ID":"5503f728-1761-4eba-905c-47d8e4bb6677","Type":"ContainerDied","Data":"3133d659a338073a56bdff2d54bce868d47aea0bfebdd46d129e8048ff4f4d1c"} Oct 30 12:28:15.057334 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:15.056954 4468 scope.go:122] "RemoveContainer" containerID="35cf38cdaf3749007196d2100518aed9251b2456e218446df0ae504149178116" Oct 30 12:28:15.063361 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:15.063340 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt" Oct 30 12:28:15.063495 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:15.063427 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt" event={"ID":"fafb0ac3-55ef-4d91-ba47-4ad9b392b988","Type":"ContainerDied","Data":"1a1853daa3176afa2aaf4170f7c6c6b483875fc0326d0865545541385a16a736"} Oct 30 12:28:15.073020 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:15.072997 4468 scope.go:122] "RemoveContainer" containerID="3624bcf3b0467397cb6e0d85953a203fbb5548c9e7cd392dd026e59ac11e2ca9" Oct 30 12:28:15.115877 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:15.115849 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q"] Oct 30 12:28:15.129030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:15.129005 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-266486-fb74bbcbd-t999q"] Oct 30 12:28:15.153810 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:15.153778 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt"] Oct 30 12:28:15.157961 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:15.157928 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-266486-994d58dff-lgvzt"] Oct 30 12:28:16.843412 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:16.843380 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22" Oct 30 12:28:17.090619 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:17.090581 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5503f728-1761-4eba-905c-47d8e4bb6677" path="/var/lib/kubelet/pods/5503f728-1761-4eba-905c-47d8e4bb6677/volumes" Oct 30 12:28:17.091041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:17.091027 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fafb0ac3-55ef-4d91-ba47-4ad9b392b988" path="/var/lib/kubelet/pods/fafb0ac3-55ef-4d91-ba47-4ad9b392b988/volumes" Oct 30 12:28:21.324106 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:21.307230 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv"] Oct 30 12:28:21.366975 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:21.366320 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" podUID="4e5e17a1-d0c2-4357-827e-0b83a118f510" containerName="maas-api" containerID="cri-o://255b0e84a79f3585df2c8f17ee023dedf7f16ee1adf4d0d4c880e9b73a70ff71" gracePeriod=30 Oct 30 12:28:21.437385 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:21.437340 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx"] Oct 30 12:28:21.437615 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:21.437586 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx" podUID="fac8a1fa-9988-466f-ad04-5a09ff02763a" containerName="payload-processing" containerID="cri-o://84a47e70938500cf9fae500cdf137f8ec314acfa5b3584b0cc1296249ce9787b" gracePeriod=30 Oct 30 12:28:21.451668 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:21.451637 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47"] Oct 30 12:28:21.451934 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:21.451870 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47" podUID="7243003a-ec74-47cd-9573-8083f2aeffd3" containerName="payload-pre-processing" containerID="cri-o://750dc96ce6c6ad0afa4a09ea1610290ac318c44a48645c12e9656752d8b7bd69" gracePeriod=30 Oct 30 12:28:21.945985 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:21.945893 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-ftmv4_910664f0-5127-40e1-b2c8-493f050457cc/payload-processing/0.log" Oct 30 12:28:21.962248 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:21.961906 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" Oct 30 12:28:22.023971 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.023935 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e5e17a1-d0c2-4357-827e-0b83a118f510-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5e17a1-d0c2-4357-827e-0b83a118f510-openshift-service-ca\") pod \"4e5e17a1-d0c2-4357-827e-0b83a118f510\" (UID: \"4e5e17a1-d0c2-4357-827e-0b83a118f510\") " Oct 30 12:28:22.024162 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.024008 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e5e17a1-d0c2-4357-827e-0b83a118f510-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e5e17a1-d0c2-4357-827e-0b83a118f510-metrics-certs\") pod \"4e5e17a1-d0c2-4357-827e-0b83a118f510\" (UID: \"4e5e17a1-d0c2-4357-827e-0b83a118f510\") " Oct 30 12:28:22.024162 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.024157 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5e17a1-d0c2-4357-827e-0b83a118f510-kube-api-access-5bp7k\" (UniqueName: \"kubernetes.io/projected/4e5e17a1-d0c2-4357-827e-0b83a118f510-kube-api-access-5bp7k\") pod \"4e5e17a1-d0c2-4357-827e-0b83a118f510\" (UID: \"4e5e17a1-d0c2-4357-827e-0b83a118f510\") " Oct 30 12:28:22.024289 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.024238 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e5e17a1-d0c2-4357-827e-0b83a118f510-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e5e17a1-d0c2-4357-827e-0b83a118f510-maas-api-tls\") pod \"4e5e17a1-d0c2-4357-827e-0b83a118f510\" (UID: \"4e5e17a1-d0c2-4357-827e-0b83a118f510\") " Oct 30 12:28:22.028637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.026516 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5e17a1-d0c2-4357-827e-0b83a118f510-openshift-service-ca" pod "4e5e17a1-d0c2-4357-827e-0b83a118f510" (UID: "4e5e17a1-d0c2-4357-827e-0b83a118f510"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:22.041628 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.041585 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5e17a1-d0c2-4357-827e-0b83a118f510-kube-api-access-5bp7k" pod "4e5e17a1-d0c2-4357-827e-0b83a118f510" (UID: "4e5e17a1-d0c2-4357-827e-0b83a118f510"). InnerVolumeSpecName "kube-api-access-5bp7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:22.043734 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.043703 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5e17a1-d0c2-4357-827e-0b83a118f510-metrics-certs" pod "4e5e17a1-d0c2-4357-827e-0b83a118f510" (UID: "4e5e17a1-d0c2-4357-827e-0b83a118f510"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:28:22.045315 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.045285 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5e17a1-d0c2-4357-827e-0b83a118f510-maas-api-tls" pod "4e5e17a1-d0c2-4357-827e-0b83a118f510" (UID: "4e5e17a1-d0c2-4357-827e-0b83a118f510"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:28:22.056214 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.056186 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx" Oct 30 12:28:22.109939 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.109887 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47" Oct 30 12:28:22.125887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.125842 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fac8a1fa-9988-466f-ad04-5a09ff02763a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac8a1fa-9988-466f-ad04-5a09ff02763a-plugins-config-volume\") pod \"fac8a1fa-9988-466f-ad04-5a09ff02763a\" (UID: \"fac8a1fa-9988-466f-ad04-5a09ff02763a\") " Oct 30 12:28:22.126105 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.126033 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac8a1fa-9988-466f-ad04-5a09ff02763a-kube-api-access-ptfwl\" (UniqueName: \"kubernetes.io/projected/fac8a1fa-9988-466f-ad04-5a09ff02763a-kube-api-access-ptfwl\") pod \"fac8a1fa-9988-466f-ad04-5a09ff02763a\" (UID: \"fac8a1fa-9988-466f-ad04-5a09ff02763a\") " Oct 30 12:28:22.126747 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.126675 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac8a1fa-9988-466f-ad04-5a09ff02763a-plugins-config-volume" pod "fac8a1fa-9988-466f-ad04-5a09ff02763a" (UID: "fac8a1fa-9988-466f-ad04-5a09ff02763a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:22.127273 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.127245 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5e17a1-d0c2-4357-827e-0b83a118f510-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:22.127273 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.127269 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e5e17a1-d0c2-4357-827e-0b83a118f510-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:22.127443 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.127282 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bp7k\" (UniqueName: \"kubernetes.io/projected/4e5e17a1-d0c2-4357-827e-0b83a118f510-kube-api-access-5bp7k\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:22.127443 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.127296 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac8a1fa-9988-466f-ad04-5a09ff02763a-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:22.127443 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.127309 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e5e17a1-d0c2-4357-827e-0b83a118f510-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:22.129600 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.129577 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac8a1fa-9988-466f-ad04-5a09ff02763a-kube-api-access-ptfwl" pod "fac8a1fa-9988-466f-ad04-5a09ff02763a" (UID: "fac8a1fa-9988-466f-ad04-5a09ff02763a"). InnerVolumeSpecName "kube-api-access-ptfwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:22.137137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.137106 4468 generic.go:356] "Generic (PLEG): container finished" podID="7243003a-ec74-47cd-9573-8083f2aeffd3" containerID="750dc96ce6c6ad0afa4a09ea1610290ac318c44a48645c12e9656752d8b7bd69" exitCode=0 Oct 30 12:28:22.137260 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.137207 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47" Oct 30 12:28:22.137260 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.137229 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47" event={"ID":"7243003a-ec74-47cd-9573-8083f2aeffd3","Type":"ContainerDied","Data":"750dc96ce6c6ad0afa4a09ea1610290ac318c44a48645c12e9656752d8b7bd69"} Oct 30 12:28:22.137377 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.137286 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47" event={"ID":"7243003a-ec74-47cd-9573-8083f2aeffd3","Type":"ContainerDied","Data":"943957dfa83a35494a083881e5037e8cb6d9f327baabd1dfa7a5c5d22d8f5e54"} Oct 30 12:28:22.137377 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.137324 4468 scope.go:122] "RemoveContainer" containerID="750dc96ce6c6ad0afa4a09ea1610290ac318c44a48645c12e9656752d8b7bd69" Oct 30 12:28:22.139226 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.139101 4468 generic.go:356] "Generic (PLEG): container finished" podID="fac8a1fa-9988-466f-ad04-5a09ff02763a" containerID="84a47e70938500cf9fae500cdf137f8ec314acfa5b3584b0cc1296249ce9787b" exitCode=0 Oct 30 12:28:22.140035 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.139747 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx" Oct 30 12:28:22.142402 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.142372 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx" event={"ID":"fac8a1fa-9988-466f-ad04-5a09ff02763a","Type":"ContainerDied","Data":"84a47e70938500cf9fae500cdf137f8ec314acfa5b3584b0cc1296249ce9787b"} Oct 30 12:28:22.142512 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.142434 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx" event={"ID":"fac8a1fa-9988-466f-ad04-5a09ff02763a","Type":"ContainerDied","Data":"695fd13496760d433ed1b8380f9605ad757dcf5d31172640eb3ab55f47298a76"} Oct 30 12:28:22.147842 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.144253 4468 generic.go:356] "Generic (PLEG): container finished" podID="4e5e17a1-d0c2-4357-827e-0b83a118f510" containerID="255b0e84a79f3585df2c8f17ee023dedf7f16ee1adf4d0d4c880e9b73a70ff71" exitCode=0 Oct 30 12:28:22.147842 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.144290 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" event={"ID":"4e5e17a1-d0c2-4357-827e-0b83a118f510","Type":"ContainerDied","Data":"255b0e84a79f3585df2c8f17ee023dedf7f16ee1adf4d0d4c880e9b73a70ff71"} Oct 30 12:28:22.147842 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.144328 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" event={"ID":"4e5e17a1-d0c2-4357-827e-0b83a118f510","Type":"ContainerDied","Data":"a26086da2d1a4bf3441730ebfc1d74e238f333e4203c92ef5566f7a802ee5a60"} Oct 30 12:28:22.147842 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.144395 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv" Oct 30 12:28:22.175319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.175292 4468 scope.go:122] "RemoveContainer" containerID="750dc96ce6c6ad0afa4a09ea1610290ac318c44a48645c12e9656752d8b7bd69" Oct 30 12:28:22.178416 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.178391 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-ftmv4_910664f0-5127-40e1-b2c8-493f050457cc/payload-processing/0.log" Oct 30 12:28:22.178609 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:28:22.178517 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750dc96ce6c6ad0afa4a09ea1610290ac318c44a48645c12e9656752d8b7bd69\": container with ID starting with 750dc96ce6c6ad0afa4a09ea1610290ac318c44a48645c12e9656752d8b7bd69 not found: ID does not exist" containerID="750dc96ce6c6ad0afa4a09ea1610290ac318c44a48645c12e9656752d8b7bd69" Oct 30 12:28:22.178609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.178550 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750dc96ce6c6ad0afa4a09ea1610290ac318c44a48645c12e9656752d8b7bd69"} err="failed to get container status \"750dc96ce6c6ad0afa4a09ea1610290ac318c44a48645c12e9656752d8b7bd69\": rpc error: code = NotFound desc = could not find container \"750dc96ce6c6ad0afa4a09ea1610290ac318c44a48645c12e9656752d8b7bd69\": container with ID starting with 750dc96ce6c6ad0afa4a09ea1610290ac318c44a48645c12e9656752d8b7bd69 not found: ID does not exist" Oct 30 12:28:22.178609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.178575 4468 scope.go:122] "RemoveContainer" containerID="84a47e70938500cf9fae500cdf137f8ec314acfa5b3584b0cc1296249ce9787b" Oct 30 12:28:22.193494 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.193469 4468 scope.go:122] "RemoveContainer" containerID="84a47e70938500cf9fae500cdf137f8ec314acfa5b3584b0cc1296249ce9787b" Oct 30 12:28:22.193851 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:28:22.193817 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84a47e70938500cf9fae500cdf137f8ec314acfa5b3584b0cc1296249ce9787b\": container with ID starting with 84a47e70938500cf9fae500cdf137f8ec314acfa5b3584b0cc1296249ce9787b not found: ID does not exist" containerID="84a47e70938500cf9fae500cdf137f8ec314acfa5b3584b0cc1296249ce9787b" Oct 30 12:28:22.193925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.193861 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a47e70938500cf9fae500cdf137f8ec314acfa5b3584b0cc1296249ce9787b"} err="failed to get container status \"84a47e70938500cf9fae500cdf137f8ec314acfa5b3584b0cc1296249ce9787b\": rpc error: code = NotFound desc = could not find container \"84a47e70938500cf9fae500cdf137f8ec314acfa5b3584b0cc1296249ce9787b\": container with ID starting with 84a47e70938500cf9fae500cdf137f8ec314acfa5b3584b0cc1296249ce9787b not found: ID does not exist" Oct 30 12:28:22.193925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.193880 4468 scope.go:122] "RemoveContainer" containerID="255b0e84a79f3585df2c8f17ee023dedf7f16ee1adf4d0d4c880e9b73a70ff71" Oct 30 12:28:22.205158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.205133 4468 scope.go:122] "RemoveContainer" containerID="255b0e84a79f3585df2c8f17ee023dedf7f16ee1adf4d0d4c880e9b73a70ff71" Oct 30 12:28:22.205503 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:28:22.205470 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255b0e84a79f3585df2c8f17ee023dedf7f16ee1adf4d0d4c880e9b73a70ff71\": container with ID starting with 255b0e84a79f3585df2c8f17ee023dedf7f16ee1adf4d0d4c880e9b73a70ff71 not found: ID does not exist" containerID="255b0e84a79f3585df2c8f17ee023dedf7f16ee1adf4d0d4c880e9b73a70ff71" Oct 30 12:28:22.205606 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.205502 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255b0e84a79f3585df2c8f17ee023dedf7f16ee1adf4d0d4c880e9b73a70ff71"} err="failed to get container status \"255b0e84a79f3585df2c8f17ee023dedf7f16ee1adf4d0d4c880e9b73a70ff71\": rpc error: code = NotFound desc = could not find container \"255b0e84a79f3585df2c8f17ee023dedf7f16ee1adf4d0d4c880e9b73a70ff71\": container with ID starting with 255b0e84a79f3585df2c8f17ee023dedf7f16ee1adf4d0d4c880e9b73a70ff71 not found: ID does not exist" Oct 30 12:28:22.228680 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.228652 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7243003a-ec74-47cd-9573-8083f2aeffd3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7243003a-ec74-47cd-9573-8083f2aeffd3-plugins-config-volume\") pod \"7243003a-ec74-47cd-9573-8083f2aeffd3\" (UID: \"7243003a-ec74-47cd-9573-8083f2aeffd3\") " Oct 30 12:28:22.228810 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.228696 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7243003a-ec74-47cd-9573-8083f2aeffd3-kube-api-access-7krnr\" (UniqueName: \"kubernetes.io/projected/7243003a-ec74-47cd-9573-8083f2aeffd3-kube-api-access-7krnr\") pod \"7243003a-ec74-47cd-9573-8083f2aeffd3\" (UID: \"7243003a-ec74-47cd-9573-8083f2aeffd3\") " Oct 30 12:28:22.229101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.229061 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7243003a-ec74-47cd-9573-8083f2aeffd3-plugins-config-volume" pod "7243003a-ec74-47cd-9573-8083f2aeffd3" (UID: "7243003a-ec74-47cd-9573-8083f2aeffd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:22.229560 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.229539 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7243003a-ec74-47cd-9573-8083f2aeffd3-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:22.229646 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.229563 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptfwl\" (UniqueName: \"kubernetes.io/projected/fac8a1fa-9988-466f-ad04-5a09ff02763a-kube-api-access-ptfwl\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:22.230887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.230871 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7243003a-ec74-47cd-9573-8083f2aeffd3-kube-api-access-7krnr" pod "7243003a-ec74-47cd-9573-8083f2aeffd3" (UID: "7243003a-ec74-47cd-9573-8083f2aeffd3"). InnerVolumeSpecName "kube-api-access-7krnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:22.249402 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.249381 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv"] Oct 30 12:28:22.254239 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.254217 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-876549-c9d95d78-sfmrv"] Oct 30 12:28:22.290727 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.290687 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx"] Oct 30 12:28:22.295618 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.295586 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-876549-9b675db4d-mppdx"] Oct 30 12:28:22.331080 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.331011 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7krnr\" (UniqueName: \"kubernetes.io/projected/7243003a-ec74-47cd-9573-8083f2aeffd3-kube-api-access-7krnr\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:22.382105 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.380412 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22_b6055421-f8de-4e98-bd0e-4f1a0cbeefc5/payload-processing/0.log" Oct 30 12:28:22.492132 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.492054 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47"] Oct 30 12:28:22.500691 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:22.500656 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-876549-6858f64d9f-4rg47"] Oct 30 12:28:23.092113 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.092081 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e5e17a1-d0c2-4357-827e-0b83a118f510" path="/var/lib/kubelet/pods/4e5e17a1-d0c2-4357-827e-0b83a118f510/volumes" Oct 30 12:28:23.092558 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.092543 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7243003a-ec74-47cd-9573-8083f2aeffd3" path="/var/lib/kubelet/pods/7243003a-ec74-47cd-9573-8083f2aeffd3/volumes" Oct 30 12:28:23.092979 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.092964 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fac8a1fa-9988-466f-ad04-5a09ff02763a" path="/var/lib/kubelet/pods/fac8a1fa-9988-466f-ad04-5a09ff02763a/volumes" Oct 30 12:28:23.238041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.237999 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6"] Oct 30 12:28:23.239629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.239599 4468 cpu_manager.go:414] "containerMap: removing container" podUID="4ad41b97-a567-4fc6-8d54-842e1547bfd7" containerName="istio-proxy" Oct 30 12:28:23.239629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.239629 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad41b97-a567-4fc6-8d54-842e1547bfd7" containerName="istio-proxy" Oct 30 12:28:23.239815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.239656 4468 cpu_manager.go:414] "containerMap: removing container" podUID="efb7c9be-be70-4b37-8657-95c9a1e283ca" containerName="istio-proxy" Oct 30 12:28:23.239815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.239664 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb7c9be-be70-4b37-8657-95c9a1e283ca" containerName="istio-proxy" Oct 30 12:28:23.239815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.239678 4468 cpu_manager.go:414] "containerMap: removing container" podUID="fac8a1fa-9988-466f-ad04-5a09ff02763a" containerName="payload-processing" Oct 30 12:28:23.239815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.239694 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac8a1fa-9988-466f-ad04-5a09ff02763a" containerName="payload-processing" Oct 30 12:28:23.239815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.239706 4468 cpu_manager.go:414] "containerMap: removing container" podUID="5503f728-1761-4eba-905c-47d8e4bb6677" containerName="payload-processing" Oct 30 12:28:23.239815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.239714 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5503f728-1761-4eba-905c-47d8e4bb6677" containerName="payload-processing" Oct 30 12:28:23.239815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.239755 4468 cpu_manager.go:414] "containerMap: removing container" podUID="fafb0ac3-55ef-4d91-ba47-4ad9b392b988" containerName="payload-pre-processing" Oct 30 12:28:23.239815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.239763 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fafb0ac3-55ef-4d91-ba47-4ad9b392b988" containerName="payload-pre-processing" Oct 30 12:28:23.239815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.239782 4468 cpu_manager.go:414] "containerMap: removing container" podUID="4e5e17a1-d0c2-4357-827e-0b83a118f510" containerName="maas-api" Oct 30 12:28:23.239815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.239789 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5e17a1-d0c2-4357-827e-0b83a118f510" containerName="maas-api" Oct 30 12:28:23.240342 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.239825 4468 cpu_manager.go:414] "containerMap: removing container" podUID="7243003a-ec74-47cd-9573-8083f2aeffd3" containerName="payload-pre-processing" Oct 30 12:28:23.240342 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.239832 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7243003a-ec74-47cd-9573-8083f2aeffd3" containerName="payload-pre-processing" Oct 30 12:28:23.240342 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.239985 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="fafb0ac3-55ef-4d91-ba47-4ad9b392b988" containerName="payload-pre-processing" Oct 30 12:28:23.240342 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.240002 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5e17a1-d0c2-4357-827e-0b83a118f510" containerName="maas-api" Oct 30 12:28:23.240342 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.240016 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="7243003a-ec74-47cd-9573-8083f2aeffd3" containerName="payload-pre-processing" Oct 30 12:28:23.240342 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.240029 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad41b97-a567-4fc6-8d54-842e1547bfd7" containerName="istio-proxy" Oct 30 12:28:23.240342 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.240037 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb7c9be-be70-4b37-8657-95c9a1e283ca" containerName="istio-proxy" Oct 30 12:28:23.240342 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.240048 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac8a1fa-9988-466f-ad04-5a09ff02763a" containerName="payload-processing" Oct 30 12:28:23.240342 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.240058 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="5503f728-1761-4eba-905c-47d8e4bb6677" containerName="payload-processing" Oct 30 12:28:23.240342 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.240085 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb7c9be-be70-4b37-8657-95c9a1e283ca" containerName="istio-proxy" Oct 30 12:28:23.246060 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.246033 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.275311 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.275280 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6"] Oct 30 12:28:23.298735 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.298624 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-ab9381-openshift-default-dockercfg-wqbb2\"" Oct 30 12:28:23.349374 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.349287 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-data\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.349374 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.349319 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-token\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.349879 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.349442 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-credential-socket\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.349879 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.349589 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjbts\" (UniqueName: \"kubernetes.io/projected/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-kube-api-access-tjbts\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.349879 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.349676 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-workload-socket\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.349879 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.349713 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istiod-ca-cert\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.349879 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.349731 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-envoy\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.349879 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.349828 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-workload-certs\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.350108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.349884 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-podinfo\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.453090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.452224 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-credential-socket\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.453090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.452376 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjbts\" (UniqueName: \"kubernetes.io/projected/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-kube-api-access-tjbts\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.453090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.452447 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-workload-socket\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.453090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.452499 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istiod-ca-cert\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.453090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.452531 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-envoy\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.453090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.452593 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-workload-certs\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.453090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.452626 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-podinfo\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.453090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.452703 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-data\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.453090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.452727 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-token\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.453715 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.453699 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-credential-socket\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.466095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.457465 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-workload-certs\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.466095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.458158 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istiod-ca-cert\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.469964 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.469914 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-data\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.491108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.491023 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-workload-socket\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.492903 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.492857 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-podinfo\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.537118 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.505985 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjbts\" (UniqueName: \"kubernetes.io/projected/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-kube-api-access-tjbts\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.537118 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.525697 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-token\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.553099 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.548465 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-envoy\") pod \"e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.614146 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:23.613677 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:23.686287 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:28:23.686257 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:28:24.031426 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:28:24.031379 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f3f4ba_2b76_43e2_905f_25dd92a8f2a4.slice/crio-ad01a896dfed75e31cd8e3ab38fe7ab76a406c63197a33bfa7713193dafff0e8 WatchSource:0}: Error finding container ad01a896dfed75e31cd8e3ab38fe7ab76a406c63197a33bfa7713193dafff0e8: Status 404 returned error can't find the container with id ad01a896dfed75e31cd8e3ab38fe7ab76a406c63197a33bfa7713193dafff0e8 Oct 30 12:28:24.034439 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.034402 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:28:24.034559 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.034502 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:28:24.034559 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.034553 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:28:24.036521 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.036492 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6"] Oct 30 12:28:24.203113 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.201159 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" event={"ID":"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4","Type":"ContainerStarted","Data":"ad01a896dfed75e31cd8e3ab38fe7ab76a406c63197a33bfa7713193dafff0e8"} Oct 30 12:28:24.879093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.874991 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj"] Oct 30 12:28:24.879093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.875984 4468 cpu_manager.go:414] "containerMap: removing container" podUID="efb7c9be-be70-4b37-8657-95c9a1e283ca" containerName="istio-proxy" Oct 30 12:28:24.879093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.876005 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb7c9be-be70-4b37-8657-95c9a1e283ca" containerName="istio-proxy" Oct 30 12:28:24.936114 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.916936 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:24.936114 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.935587 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8e3112\"/\"openshift-service-ca.crt\"" Oct 30 12:28:24.939501 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.939472 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj"] Oct 30 12:28:24.948849 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.948570 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8e3112\"/\"kube-root-ca.crt\"" Oct 30 12:28:24.948849 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.948768 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8e3112\"/\"ipp-route-8e3112-kserve-self-signed-certs\"" Oct 30 12:28:24.948849 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.948836 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8e3112\"/\"default-dockercfg-fz7jx\"" Oct 30 12:28:24.999389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.999340 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j6bj\" (UniqueName: \"kubernetes.io/projected/17be3a09-0989-4ccb-ae91-4a316e0fdca8-kube-api-access-2j6bj\") pod \"ipp-route-8e3112-kserve-7c789474df-fnfmj\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:24.999575 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.999499 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-home\") pod \"ipp-route-8e3112-kserve-7c789474df-fnfmj\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:24.999649 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.999569 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-model-cache\") pod \"ipp-route-8e3112-kserve-7c789474df-fnfmj\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:24.999715 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.999677 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-dshm\") pod \"ipp-route-8e3112-kserve-7c789474df-fnfmj\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:24.999820 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.999798 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17be3a09-0989-4ccb-ae91-4a316e0fdca8-tls-certs\") pod \"ipp-route-8e3112-kserve-7c789474df-fnfmj\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:24.999889 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:24.999877 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-tmp-dir\") pod \"ipp-route-8e3112-kserve-7c789474df-fnfmj\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:25.136364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:25.127973 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j6bj\" (UniqueName: \"kubernetes.io/projected/17be3a09-0989-4ccb-ae91-4a316e0fdca8-kube-api-access-2j6bj\") pod \"ipp-route-8e3112-kserve-7c789474df-fnfmj\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:25.136364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:25.128109 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-home\") pod \"ipp-route-8e3112-kserve-7c789474df-fnfmj\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:25.136364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:25.128162 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-model-cache\") pod \"ipp-route-8e3112-kserve-7c789474df-fnfmj\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:25.136364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:25.128205 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-dshm\") pod \"ipp-route-8e3112-kserve-7c789474df-fnfmj\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:25.136364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:25.128329 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17be3a09-0989-4ccb-ae91-4a316e0fdca8-tls-certs\") pod \"ipp-route-8e3112-kserve-7c789474df-fnfmj\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:25.136364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:25.128419 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-tmp-dir\") pod \"ipp-route-8e3112-kserve-7c789474df-fnfmj\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:25.136364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:25.128920 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-tmp-dir\") pod \"ipp-route-8e3112-kserve-7c789474df-fnfmj\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:25.136364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:25.130355 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-home\") pod \"ipp-route-8e3112-kserve-7c789474df-fnfmj\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:25.136364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:25.130593 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-model-cache\") pod \"ipp-route-8e3112-kserve-7c789474df-fnfmj\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:25.155844 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:25.155777 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17be3a09-0989-4ccb-ae91-4a316e0fdca8-tls-certs\") pod \"ipp-route-8e3112-kserve-7c789474df-fnfmj\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:25.224309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:25.204392 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-dshm\") pod \"ipp-route-8e3112-kserve-7c789474df-fnfmj\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:25.255244 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:25.255209 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j6bj\" (UniqueName: \"kubernetes.io/projected/17be3a09-0989-4ccb-ae91-4a316e0fdca8-kube-api-access-2j6bj\") pod \"ipp-route-8e3112-kserve-7c789474df-fnfmj\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:25.289341 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:25.288182 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" event={"ID":"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4","Type":"ContainerStarted","Data":"660bc54ecfc8ebbf9214fd099a98bf6167e1e0349d549ca65da2d010150c4b5b"} Oct 30 12:28:25.341975 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:25.341914 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" podStartSLOduration=2.341898962 podStartE2EDuration="2.341898962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:25.338009266 +0000 UTC m=+2056.372147662" watchObservedRunningTime="2026-10-30 12:28:25.341898962 +0000 UTC m=+2056.376037358" Oct 30 12:28:25.552254 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:25.552219 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:25.614989 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:25.614963 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:25.622590 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:25.622560 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:25.889057 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:28:25.889027 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17be3a09_0989_4ccb_ae91_4a316e0fdca8.slice/crio-50801ba96b679ff6ea420a3c2b146d4faea5ed1b8357300703eed9dcb1f6a02d WatchSource:0}: Error finding container 50801ba96b679ff6ea420a3c2b146d4faea5ed1b8357300703eed9dcb1f6a02d: Status 404 returned error can't find the container with id 50801ba96b679ff6ea420a3c2b146d4faea5ed1b8357300703eed9dcb1f6a02d Oct 30 12:28:25.891847 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:25.891818 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj"] Oct 30 12:28:26.295913 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:26.295875 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" event={"ID":"17be3a09-0989-4ccb-ae91-4a316e0fdca8","Type":"ContainerStarted","Data":"47ead5edb7c82b5ed3641c1e132e7e366a5022f7c7e151d5b3f7cdaa8b0bc6d5"} Oct 30 12:28:26.296146 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:26.295923 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" event={"ID":"17be3a09-0989-4ccb-ae91-4a316e0fdca8","Type":"ContainerStarted","Data":"50801ba96b679ff6ea420a3c2b146d4faea5ed1b8357300703eed9dcb1f6a02d"} Oct 30 12:28:26.296232 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:26.296198 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:26.297334 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:26.297312 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:28:26.358049 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:26.357987 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" podStartSLOduration=2.357973549 podStartE2EDuration="2.357973549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:26.354629991 +0000 UTC m=+2057.388768387" watchObservedRunningTime="2026-10-30 12:28:26.357973549 +0000 UTC m=+2057.392111944" Oct 30 12:28:26.574044 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:26.573958 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:26.932683 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:26.932637 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd"] Oct 30 12:28:29.934881 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:29.934842 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn"] Oct 30 12:28:29.939551 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:29.939527 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn" Oct 30 12:28:29.959214 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:29.959178 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn"] Oct 30 12:28:29.965482 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:29.965451 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-ab9381\"" Oct 30 12:28:29.999787 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:29.999734 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8"] Oct 30 12:28:30.040042 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:30.039999 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8" Oct 30 12:28:30.040728 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:30.040677 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8"] Oct 30 12:28:30.104098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:30.102710 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-ab9381-dockercfg-67dm5\"" Oct 30 12:28:30.108096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:30.104883 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5vbc\" (UniqueName: \"kubernetes.io/projected/24ab8c1a-3da3-48a4-97ef-fed9c22f14ed-kube-api-access-n5vbc\") pod \"payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8\" (UID: \"24ab8c1a-3da3-48a4-97ef-fed9c22f14ed\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8" Oct 30 12:28:30.108096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:30.104979 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbt7r\" (UniqueName: \"kubernetes.io/projected/16b5b939-9d39-4175-9735-25bd51175c3d-kube-api-access-gbt7r\") pod \"payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn\" (UID: \"16b5b939-9d39-4175-9735-25bd51175c3d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn" Oct 30 12:28:30.113097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:30.112164 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b5b939-9d39-4175-9735-25bd51175c3d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn\" (UID: \"16b5b939-9d39-4175-9735-25bd51175c3d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn" Oct 30 12:28:30.113097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:30.112322 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ab8c1a-3da3-48a4-97ef-fed9c22f14ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8\" (UID: \"24ab8c1a-3da3-48a4-97ef-fed9c22f14ed\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8" Oct 30 12:28:30.258147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:30.229625 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b5b939-9d39-4175-9735-25bd51175c3d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn\" (UID: \"16b5b939-9d39-4175-9735-25bd51175c3d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn" Oct 30 12:28:30.258147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:30.229742 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ab8c1a-3da3-48a4-97ef-fed9c22f14ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8\" (UID: \"24ab8c1a-3da3-48a4-97ef-fed9c22f14ed\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8" Oct 30 12:28:30.258147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:30.229869 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5vbc\" (UniqueName: \"kubernetes.io/projected/24ab8c1a-3da3-48a4-97ef-fed9c22f14ed-kube-api-access-n5vbc\") pod \"payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8\" (UID: \"24ab8c1a-3da3-48a4-97ef-fed9c22f14ed\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8" Oct 30 12:28:30.258147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:30.229958 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbt7r\" (UniqueName: \"kubernetes.io/projected/16b5b939-9d39-4175-9735-25bd51175c3d-kube-api-access-gbt7r\") pod \"payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn\" (UID: \"16b5b939-9d39-4175-9735-25bd51175c3d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn" Oct 30 12:28:30.260744 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:30.260495 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ab8c1a-3da3-48a4-97ef-fed9c22f14ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8\" (UID: \"24ab8c1a-3da3-48a4-97ef-fed9c22f14ed\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8" Oct 30 12:28:30.261442 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:30.261413 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b5b939-9d39-4175-9735-25bd51175c3d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn\" (UID: \"16b5b939-9d39-4175-9735-25bd51175c3d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn" Oct 30 12:28:30.361094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:30.338518 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbt7r\" (UniqueName: \"kubernetes.io/projected/16b5b939-9d39-4175-9735-25bd51175c3d-kube-api-access-gbt7r\") pod \"payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn\" (UID: \"16b5b939-9d39-4175-9735-25bd51175c3d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn" Oct 30 12:28:30.362716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:30.362646 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5vbc\" (UniqueName: \"kubernetes.io/projected/24ab8c1a-3da3-48a4-97ef-fed9c22f14ed-kube-api-access-n5vbc\") pod \"payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8\" (UID: \"24ab8c1a-3da3-48a4-97ef-fed9c22f14ed\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8" Oct 30 12:28:30.559460 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:30.558183 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn" Oct 30 12:28:30.560918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:30.560894 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8" Oct 30 12:28:30.602563 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:30.602530 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd"] Oct 30 12:28:30.602893 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:30.602848 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" podUID="51896ebb-81a0-4746-a5a1-22521ebdc85b" containerName="istio-proxy" containerID="cri-o://510ace7d7570fbbaf22c543c9649da10aa286520d709e5d332c6be0d6c505d35" gracePeriod=30 Oct 30 12:28:31.205779 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:31.203606 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8"] Oct 30 12:28:31.205779 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:31.203666 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn"] Oct 30 12:28:31.650435 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:31.639710 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn" event={"ID":"16b5b939-9d39-4175-9735-25bd51175c3d","Type":"ContainerStarted","Data":"eb8b0f85955969034f41eaee1287e1a0a88277b8500d92d0104934861c783ab8"} Oct 30 12:28:31.659744 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:31.659702 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8" event={"ID":"24ab8c1a-3da3-48a4-97ef-fed9c22f14ed","Type":"ContainerStarted","Data":"8b6e0ddc4e3b70e0505c1358eacc4dcd65bd99e1f0a0db1b43a17ce611481baa"} Oct 30 12:28:31.968567 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:31.934134 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd" podUID="ee03fe8b-948f-4a5d-90c0-14c97a9a71f5" containerName="payload-processing" containerID="cri-o://3feb0a78f49faf3317172fe4bb632b4b598e84224159abcdfa9a75867ac13f11" gracePeriod=25 Oct 30 12:28:32.313359 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.313326 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd" Oct 30 12:28:32.498191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.498155 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee03fe8b-948f-4a5d-90c0-14c97a9a71f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee03fe8b-948f-4a5d-90c0-14c97a9a71f5-plugins-config-volume\") pod \"ee03fe8b-948f-4a5d-90c0-14c97a9a71f5\" (UID: \"ee03fe8b-948f-4a5d-90c0-14c97a9a71f5\") " Oct 30 12:28:32.498362 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.498208 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee03fe8b-948f-4a5d-90c0-14c97a9a71f5-kube-api-access-dvkht\" (UniqueName: \"kubernetes.io/projected/ee03fe8b-948f-4a5d-90c0-14c97a9a71f5-kube-api-access-dvkht\") pod \"ee03fe8b-948f-4a5d-90c0-14c97a9a71f5\" (UID: \"ee03fe8b-948f-4a5d-90c0-14c97a9a71f5\") " Oct 30 12:28:32.498527 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.498500 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee03fe8b-948f-4a5d-90c0-14c97a9a71f5-plugins-config-volume" pod "ee03fe8b-948f-4a5d-90c0-14c97a9a71f5" (UID: "ee03fe8b-948f-4a5d-90c0-14c97a9a71f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:32.498984 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.498963 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee03fe8b-948f-4a5d-90c0-14c97a9a71f5-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:32.500324 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.500304 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee03fe8b-948f-4a5d-90c0-14c97a9a71f5-kube-api-access-dvkht" pod "ee03fe8b-948f-4a5d-90c0-14c97a9a71f5" (UID: "ee03fe8b-948f-4a5d-90c0-14c97a9a71f5"). InnerVolumeSpecName "kube-api-access-dvkht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:32.600615 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.600570 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvkht\" (UniqueName: \"kubernetes.io/projected/ee03fe8b-948f-4a5d-90c0-14c97a9a71f5-kube-api-access-dvkht\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:32.668756 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.668713 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8" event={"ID":"24ab8c1a-3da3-48a4-97ef-fed9c22f14ed","Type":"ContainerStarted","Data":"255c955e1f4ec668706140d69abe8627ff892878d8ac7d82b2303cdcebe2ed45"} Oct 30 12:28:32.668949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.668788 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8" Oct 30 12:28:32.670309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.670283 4468 generic.go:356] "Generic (PLEG): container finished" podID="ee03fe8b-948f-4a5d-90c0-14c97a9a71f5" containerID="3feb0a78f49faf3317172fe4bb632b4b598e84224159abcdfa9a75867ac13f11" exitCode=0 Oct 30 12:28:32.670425 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.670319 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd" Oct 30 12:28:32.670425 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.670342 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd" event={"ID":"ee03fe8b-948f-4a5d-90c0-14c97a9a71f5","Type":"ContainerDied","Data":"3feb0a78f49faf3317172fe4bb632b4b598e84224159abcdfa9a75867ac13f11"} Oct 30 12:28:32.670425 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.670385 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd" event={"ID":"ee03fe8b-948f-4a5d-90c0-14c97a9a71f5","Type":"ContainerDied","Data":"362795c150d4a111ef1f55bdbdcd8b2265dbbd1fe5772db7c955dd8561f85124"} Oct 30 12:28:32.670425 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.670398 4468 scope.go:122] "RemoveContainer" containerID="3feb0a78f49faf3317172fe4bb632b4b598e84224159abcdfa9a75867ac13f11" Oct 30 12:28:32.672346 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.672309 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn" event={"ID":"16b5b939-9d39-4175-9735-25bd51175c3d","Type":"ContainerStarted","Data":"ad425d39f0b59a984f85bb145f7587f28a28c8e98ab2ce669e7b9b9cbbe52171"} Oct 30 12:28:32.672477 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.672460 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn" Oct 30 12:28:32.685332 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.685309 4468 scope.go:122] "RemoveContainer" containerID="3feb0a78f49faf3317172fe4bb632b4b598e84224159abcdfa9a75867ac13f11" Oct 30 12:28:32.685697 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:28:32.685672 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3feb0a78f49faf3317172fe4bb632b4b598e84224159abcdfa9a75867ac13f11\": container with ID starting with 3feb0a78f49faf3317172fe4bb632b4b598e84224159abcdfa9a75867ac13f11 not found: ID does not exist" containerID="3feb0a78f49faf3317172fe4bb632b4b598e84224159abcdfa9a75867ac13f11" Oct 30 12:28:32.685768 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.685707 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3feb0a78f49faf3317172fe4bb632b4b598e84224159abcdfa9a75867ac13f11"} err="failed to get container status \"3feb0a78f49faf3317172fe4bb632b4b598e84224159abcdfa9a75867ac13f11\": rpc error: code = NotFound desc = could not find container \"3feb0a78f49faf3317172fe4bb632b4b598e84224159abcdfa9a75867ac13f11\": container with ID starting with 3feb0a78f49faf3317172fe4bb632b4b598e84224159abcdfa9a75867ac13f11 not found: ID does not exist" Oct 30 12:28:32.712981 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.712926 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8" podStartSLOduration=3.712913391 podStartE2EDuration="3.712913391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:32.711245401 +0000 UTC m=+2063.745383800" watchObservedRunningTime="2026-10-30 12:28:32.712913391 +0000 UTC m=+2063.747051787" Oct 30 12:28:32.764331 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.764223 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn" podStartSLOduration=3.764199343 podStartE2EDuration="3.764199343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:32.760364155 +0000 UTC m=+2063.794502552" watchObservedRunningTime="2026-10-30 12:28:32.764199343 +0000 UTC m=+2063.798337739" Oct 30 12:28:32.802996 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.802954 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd"] Oct 30 12:28:32.807533 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:32.807502 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-6648d989f8-2mdgd"] Oct 30 12:28:33.092189 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:33.092112 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee03fe8b-948f-4a5d-90c0-14c97a9a71f5" path="/var/lib/kubelet/pods/ee03fe8b-948f-4a5d-90c0-14c97a9a71f5/volumes" Oct 30 12:28:33.336214 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:33.336170 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:28:33.336688 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:33.336242 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" podUID="51896ebb-81a0-4746-a5a1-22521ebdc85b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:28:35.552703 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:35.552665 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:35.566277 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:35.566245 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:35.708429 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:35.708397 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:28:35.989674 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:35.989648 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:28:36.065210 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.065171 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-workload-socket\") pod \"51896ebb-81a0-4746-a5a1-22521ebdc85b\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " Oct 30 12:28:36.065572 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.065462 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-workload-socket" pod "51896ebb-81a0-4746-a5a1-22521ebdc85b" (UID: "51896ebb-81a0-4746-a5a1-22521ebdc85b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:36.065687 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.065674 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-envoy\") pod \"51896ebb-81a0-4746-a5a1-22521ebdc85b\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " Oct 30 12:28:36.066116 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.066028 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-token\" (UniqueName: \"kubernetes.io/projected/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-token\") pod \"51896ebb-81a0-4746-a5a1-22521ebdc85b\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " Oct 30 12:28:36.066233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.066196 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51896ebb-81a0-4746-a5a1-22521ebdc85b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51896ebb-81a0-4746-a5a1-22521ebdc85b-istiod-ca-cert\") pod \"51896ebb-81a0-4746-a5a1-22521ebdc85b\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " Oct 30 12:28:36.066306 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.066236 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-podinfo\") pod \"51896ebb-81a0-4746-a5a1-22521ebdc85b\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " Oct 30 12:28:36.066788 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.066361 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-credential-socket\") pod \"51896ebb-81a0-4746-a5a1-22521ebdc85b\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " Oct 30 12:28:36.066788 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.066743 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51896ebb-81a0-4746-a5a1-22521ebdc85b-istiod-ca-cert" pod "51896ebb-81a0-4746-a5a1-22521ebdc85b" (UID: "51896ebb-81a0-4746-a5a1-22521ebdc85b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:36.067166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.067146 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-workload-certs\") pod \"51896ebb-81a0-4746-a5a1-22521ebdc85b\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " Oct 30 12:28:36.067377 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.067241 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-credential-socket" pod "51896ebb-81a0-4746-a5a1-22521ebdc85b" (UID: "51896ebb-81a0-4746-a5a1-22521ebdc85b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:36.067458 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.067433 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-workload-certs" pod "51896ebb-81a0-4746-a5a1-22521ebdc85b" (UID: "51896ebb-81a0-4746-a5a1-22521ebdc85b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:36.067560 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.067543 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51896ebb-81a0-4746-a5a1-22521ebdc85b-kube-api-access-fx2s4\" (UniqueName: \"kubernetes.io/projected/51896ebb-81a0-4746-a5a1-22521ebdc85b-kube-api-access-fx2s4\") pod \"51896ebb-81a0-4746-a5a1-22521ebdc85b\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " Oct 30 12:28:36.067719 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.067703 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-data\") pod \"51896ebb-81a0-4746-a5a1-22521ebdc85b\" (UID: \"51896ebb-81a0-4746-a5a1-22521ebdc85b\") " Oct 30 12:28:36.068035 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.068006 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-data" pod "51896ebb-81a0-4746-a5a1-22521ebdc85b" (UID: "51896ebb-81a0-4746-a5a1-22521ebdc85b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:36.068362 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.068327 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-envoy" pod "51896ebb-81a0-4746-a5a1-22521ebdc85b" (UID: "51896ebb-81a0-4746-a5a1-22521ebdc85b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:36.068672 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.068648 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:36.068767 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.068679 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:36.068767 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.068695 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:36.068767 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.068710 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51896ebb-81a0-4746-a5a1-22521ebdc85b-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:36.068767 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.068723 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:36.068767 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.068737 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51896ebb-81a0-4746-a5a1-22521ebdc85b-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:36.069692 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.069436 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-podinfo" pod "51896ebb-81a0-4746-a5a1-22521ebdc85b" (UID: "51896ebb-81a0-4746-a5a1-22521ebdc85b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:28:36.069794 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.069769 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51896ebb-81a0-4746-a5a1-22521ebdc85b-kube-api-access-fx2s4" pod "51896ebb-81a0-4746-a5a1-22521ebdc85b" (UID: "51896ebb-81a0-4746-a5a1-22521ebdc85b"). InnerVolumeSpecName "kube-api-access-fx2s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:36.071317 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.071287 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-token" pod "51896ebb-81a0-4746-a5a1-22521ebdc85b" (UID: "51896ebb-81a0-4746-a5a1-22521ebdc85b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:36.171739 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.171637 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx2s4\" (UniqueName: \"kubernetes.io/projected/51896ebb-81a0-4746-a5a1-22521ebdc85b-kube-api-access-fx2s4\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:36.171739 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.171685 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:36.171739 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.171700 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51896ebb-81a0-4746-a5a1-22521ebdc85b-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:28:36.705701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.705662 4468 generic.go:356] "Generic (PLEG): container finished" podID="51896ebb-81a0-4746-a5a1-22521ebdc85b" containerID="510ace7d7570fbbaf22c543c9649da10aa286520d709e5d332c6be0d6c505d35" exitCode=0 Oct 30 12:28:36.705701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.705710 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" Oct 30 12:28:36.706289 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.705743 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" event={"ID":"51896ebb-81a0-4746-a5a1-22521ebdc85b","Type":"ContainerDied","Data":"510ace7d7570fbbaf22c543c9649da10aa286520d709e5d332c6be0d6c505d35"} Oct 30 12:28:36.706289 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.705802 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd" event={"ID":"51896ebb-81a0-4746-a5a1-22521ebdc85b","Type":"ContainerDied","Data":"f7880b2f9309690929f57453df1b1004b7c0acb99b7c728e43d212ee4a26c12c"} Oct 30 12:28:36.706289 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.705832 4468 scope.go:122] "RemoveContainer" containerID="510ace7d7570fbbaf22c543c9649da10aa286520d709e5d332c6be0d6c505d35" Oct 30 12:28:36.707839 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.707813 4468 generic.go:356] "Generic (PLEG): container finished" podID="3acebec4-84f8-415a-84f7-f8f7d9391a85" containerID="8013cb7d7a25a3866bb4699d4e861a05650bbb4e55c1fcffac5d0b78b20174e9" exitCode=0 Oct 30 12:28:36.707963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.707882 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" event={"ID":"3acebec4-84f8-415a-84f7-f8f7d9391a85","Type":"ContainerDied","Data":"8013cb7d7a25a3866bb4699d4e861a05650bbb4e55c1fcffac5d0b78b20174e9"} Oct 30 12:28:36.708553 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.708533 4468 scope.go:122] "RemoveContainer" containerID="8013cb7d7a25a3866bb4699d4e861a05650bbb4e55c1fcffac5d0b78b20174e9" Oct 30 12:28:36.710190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.710164 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:28:36.710287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.710230 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:28:36.710287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.710261 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:28:36.718434 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.718414 4468 scope.go:122] "RemoveContainer" containerID="dc87487d2ae8cdf7ccad88fedc84bc2181ec547cd7897bc4a57b2c122dc93ec1" Oct 30 12:28:36.728218 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.728191 4468 scope.go:122] "RemoveContainer" containerID="510ace7d7570fbbaf22c543c9649da10aa286520d709e5d332c6be0d6c505d35" Oct 30 12:28:36.728587 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:28:36.728562 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510ace7d7570fbbaf22c543c9649da10aa286520d709e5d332c6be0d6c505d35\": container with ID starting with 510ace7d7570fbbaf22c543c9649da10aa286520d709e5d332c6be0d6c505d35 not found: ID does not exist" containerID="510ace7d7570fbbaf22c543c9649da10aa286520d709e5d332c6be0d6c505d35" Oct 30 12:28:36.728693 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.728596 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510ace7d7570fbbaf22c543c9649da10aa286520d709e5d332c6be0d6c505d35"} err="failed to get container status \"510ace7d7570fbbaf22c543c9649da10aa286520d709e5d332c6be0d6c505d35\": rpc error: code = NotFound desc = could not find container \"510ace7d7570fbbaf22c543c9649da10aa286520d709e5d332c6be0d6c505d35\": container with ID starting with 510ace7d7570fbbaf22c543c9649da10aa286520d709e5d332c6be0d6c505d35 not found: ID does not exist" Oct 30 12:28:36.728693 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.728616 4468 scope.go:122] "RemoveContainer" containerID="dc87487d2ae8cdf7ccad88fedc84bc2181ec547cd7897bc4a57b2c122dc93ec1" Oct 30 12:28:36.728928 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:28:36.728908 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc87487d2ae8cdf7ccad88fedc84bc2181ec547cd7897bc4a57b2c122dc93ec1\": container with ID starting with dc87487d2ae8cdf7ccad88fedc84bc2181ec547cd7897bc4a57b2c122dc93ec1 not found: ID does not exist" containerID="dc87487d2ae8cdf7ccad88fedc84bc2181ec547cd7897bc4a57b2c122dc93ec1" Oct 30 12:28:36.728977 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.728935 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc87487d2ae8cdf7ccad88fedc84bc2181ec547cd7897bc4a57b2c122dc93ec1"} err="failed to get container status \"dc87487d2ae8cdf7ccad88fedc84bc2181ec547cd7897bc4a57b2c122dc93ec1\": rpc error: code = NotFound desc = could not find container \"dc87487d2ae8cdf7ccad88fedc84bc2181ec547cd7897bc4a57b2c122dc93ec1\": container with ID starting with dc87487d2ae8cdf7ccad88fedc84bc2181ec547cd7897bc4a57b2c122dc93ec1 not found: ID does not exist" Oct 30 12:28:36.780664 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.780637 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd"] Oct 30 12:28:36.793251 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:36.793218 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-876549-openshift-default-749d786c77-rbgzd"] Oct 30 12:28:37.097488 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:37.097405 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51896ebb-81a0-4746-a5a1-22521ebdc85b" path="/var/lib/kubelet/pods/51896ebb-81a0-4746-a5a1-22521ebdc85b/volumes" Oct 30 12:28:37.737398 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:37.737363 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" event={"ID":"3acebec4-84f8-415a-84f7-f8f7d9391a85","Type":"ContainerStarted","Data":"7971141013ff3fff8cb233b410257fde691a6ba8ec23dc0620567c64f830b2eb"} Oct 30 12:28:38.558340 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:38.558301 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:28:38.559508 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:38.559487 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 12:28:38.559587 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:38.559531 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" podUID="3acebec4-84f8-415a-84f7-f8f7d9391a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 12:28:39.558447 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:39.558407 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 12:28:39.558978 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:39.558474 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" podUID="3acebec4-84f8-415a-84f7-f8f7d9391a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 12:28:40.153902 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:40.153867 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:28:40.558516 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:40.558479 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 12:28:40.558985 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:40.558542 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" podUID="3acebec4-84f8-415a-84f7-f8f7d9391a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 12:28:41.558344 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:41.558308 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 12:28:41.558521 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:41.558362 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" podUID="3acebec4-84f8-415a-84f7-f8f7d9391a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 12:28:41.846227 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:41.846147 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs"] Oct 30 12:28:41.847705 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:41.847681 4468 cpu_manager.go:414] "containerMap: removing container" podUID="51896ebb-81a0-4746-a5a1-22521ebdc85b" containerName="istio-proxy" Oct 30 12:28:41.848053 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:41.848038 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51896ebb-81a0-4746-a5a1-22521ebdc85b" containerName="istio-proxy" Oct 30 12:28:41.848194 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:41.848183 4468 cpu_manager.go:414] "containerMap: removing container" podUID="51896ebb-81a0-4746-a5a1-22521ebdc85b" containerName="istio-proxy" Oct 30 12:28:41.848271 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:41.848262 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51896ebb-81a0-4746-a5a1-22521ebdc85b" containerName="istio-proxy" Oct 30 12:28:41.848349 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:41.848340 4468 cpu_manager.go:414] "containerMap: removing container" podUID="ee03fe8b-948f-4a5d-90c0-14c97a9a71f5" containerName="payload-processing" Oct 30 12:28:41.848424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:41.848415 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee03fe8b-948f-4a5d-90c0-14c97a9a71f5" containerName="payload-processing" Oct 30 12:28:41.848700 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:41.848685 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="51896ebb-81a0-4746-a5a1-22521ebdc85b" containerName="istio-proxy" Oct 30 12:28:41.848806 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:41.848795 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="51896ebb-81a0-4746-a5a1-22521ebdc85b" containerName="istio-proxy" Oct 30 12:28:41.848894 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:41.848884 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee03fe8b-948f-4a5d-90c0-14c97a9a71f5" containerName="payload-processing" Oct 30 12:28:41.858530 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:41.858483 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs"] Oct 30 12:28:41.858773 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:41.858651 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:41.876530 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:41.876315 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-41bdda-openshift-default-dockercfg-j9hg8\"" Oct 30 12:28:42.059937 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.059889 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-podinfo\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.060259 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.060235 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-workload-socket\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.060440 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.060424 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-envoy\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.060555 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.060539 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-token\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.060750 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.060734 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-credential-socket\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.060858 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.060844 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-workload-certs\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.097318 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.097223 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpl8b\" (UniqueName: \"kubernetes.io/projected/0166c51a-8d66-47e5-ba9a-2fb14b342215-kube-api-access-qpl8b\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.097605 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.097582 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0166c51a-8d66-47e5-ba9a-2fb14b342215-istiod-ca-cert\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.097791 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.097772 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-data\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.200130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.199097 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-token\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.200130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.199249 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-credential-socket\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.200130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.199354 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-workload-certs\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.200130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.199629 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpl8b\" (UniqueName: \"kubernetes.io/projected/0166c51a-8d66-47e5-ba9a-2fb14b342215-kube-api-access-qpl8b\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.200130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.199670 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0166c51a-8d66-47e5-ba9a-2fb14b342215-istiod-ca-cert\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.200130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.199778 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-data\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.200130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.199855 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-podinfo\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.200130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.199926 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-workload-socket\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.200130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.200042 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-envoy\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.204350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.204318 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0166c51a-8d66-47e5-ba9a-2fb14b342215-istiod-ca-cert\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.204607 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.204372 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-workload-certs\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.205064 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.205042 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-data\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.206449 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.206421 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-workload-socket\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.211018 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.210988 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-credential-socket\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.238190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.232687 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-podinfo\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.238760 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.238721 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-envoy\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.271272 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.271239 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpl8b\" (UniqueName: \"kubernetes.io/projected/0166c51a-8d66-47e5-ba9a-2fb14b342215-kube-api-access-qpl8b\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.271593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.271574 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-token\") pod \"e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.481578 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.481538 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:42.558440 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.558402 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 12:28:42.558624 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.558476 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" podUID="3acebec4-84f8-415a-84f7-f8f7d9391a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 12:28:42.838431 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:28:42.838382 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0166c51a_8d66_47e5_ba9a_2fb14b342215.slice/crio-3d10a592fb143c71f3863b73889f379e6b9743435a55e5bae358fc2dded20d5e WatchSource:0}: Error finding container 3d10a592fb143c71f3863b73889f379e6b9743435a55e5bae358fc2dded20d5e: Status 404 returned error can't find the container with id 3d10a592fb143c71f3863b73889f379e6b9743435a55e5bae358fc2dded20d5e Oct 30 12:28:42.838832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.838801 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs"] Oct 30 12:28:42.842631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.842593 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:28:42.842749 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.842680 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:28:42.842749 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:42.842726 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:28:43.562993 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:43.562961 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:28:43.564190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:43.564168 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:28:43.686009 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:43.685978 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8" Oct 30 12:28:43.688241 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:43.688212 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn" Oct 30 12:28:43.788139 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:43.788104 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" event={"ID":"0166c51a-8d66-47e5-ba9a-2fb14b342215","Type":"ContainerStarted","Data":"63d2a29bfbed3a8d0e2aad6a1b5d57f37cf82702804e1b3638fda411f6f17eb4"} Oct 30 12:28:43.788394 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:43.788368 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" event={"ID":"0166c51a-8d66-47e5-ba9a-2fb14b342215","Type":"ContainerStarted","Data":"3d10a592fb143c71f3863b73889f379e6b9743435a55e5bae358fc2dded20d5e"} Oct 30 12:28:43.837556 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:43.837456 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" podStartSLOduration=2.837440616 podStartE2EDuration="2.837440616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:43.836472224 +0000 UTC m=+2074.870610619" watchObservedRunningTime="2026-10-30 12:28:43.837440616 +0000 UTC m=+2074.871579019" Oct 30 12:28:44.398704 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:44.398666 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46"] Oct 30 12:28:44.402131 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:44.402104 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46" Oct 30 12:28:44.418496 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:44.418465 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7eee3b\"" Oct 30 12:28:44.432773 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:44.432716 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7eee3b-dockercfg-klfkm\"" Oct 30 12:28:44.440927 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:44.440879 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46"] Oct 30 12:28:44.482524 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:44.482481 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:44.489000 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:44.488967 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:44.501011 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:44.500975 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rhws\" (UniqueName: \"kubernetes.io/projected/d11d3c04-4239-41d4-a099-3c6f0d287eaf-kube-api-access-9rhws\") pod \"payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46\" (UID: \"d11d3c04-4239-41d4-a099-3c6f0d287eaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46" Oct 30 12:28:44.501232 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:44.501118 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11d3c04-4239-41d4-a099-3c6f0d287eaf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46\" (UID: \"d11d3c04-4239-41d4-a099-3c6f0d287eaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46" Oct 30 12:28:44.602929 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:44.602886 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rhws\" (UniqueName: \"kubernetes.io/projected/d11d3c04-4239-41d4-a099-3c6f0d287eaf-kube-api-access-9rhws\") pod \"payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46\" (UID: \"d11d3c04-4239-41d4-a099-3c6f0d287eaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46" Oct 30 12:28:44.603366 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:44.602977 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11d3c04-4239-41d4-a099-3c6f0d287eaf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46\" (UID: \"d11d3c04-4239-41d4-a099-3c6f0d287eaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46" Oct 30 12:28:44.603969 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:44.603944 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11d3c04-4239-41d4-a099-3c6f0d287eaf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46\" (UID: \"d11d3c04-4239-41d4-a099-3c6f0d287eaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46" Oct 30 12:28:44.628033 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:44.627990 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rhws\" (UniqueName: \"kubernetes.io/projected/d11d3c04-4239-41d4-a099-3c6f0d287eaf-kube-api-access-9rhws\") pod \"payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46\" (UID: \"d11d3c04-4239-41d4-a099-3c6f0d287eaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46" Oct 30 12:28:44.718205 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:44.718173 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46" Oct 30 12:28:44.800426 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:44.800375 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:44.802380 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:44.802351 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:28:45.061717 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:45.061628 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46"] Oct 30 12:28:45.067555 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:28:45.067505 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11d3c04_4239_41d4_a099_3c6f0d287eaf.slice/crio-ce552fe8f01287d9f56ab9eef9f59ef13d2ed41ee84eea1186eaed06fb875664 WatchSource:0}: Error finding container ce552fe8f01287d9f56ab9eef9f59ef13d2ed41ee84eea1186eaed06fb875664: Status 404 returned error can't find the container with id ce552fe8f01287d9f56ab9eef9f59ef13d2ed41ee84eea1186eaed06fb875664 Oct 30 12:28:45.828582 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:45.828542 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46" event={"ID":"d11d3c04-4239-41d4-a099-3c6f0d287eaf","Type":"ContainerStarted","Data":"0da1af733c106e8aa2bd0db13a67b82374ffa968846291e2afa32aa1e37cf3f4"} Oct 30 12:28:45.829578 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:45.829550 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46" event={"ID":"d11d3c04-4239-41d4-a099-3c6f0d287eaf","Type":"ContainerStarted","Data":"ce552fe8f01287d9f56ab9eef9f59ef13d2ed41ee84eea1186eaed06fb875664"} Oct 30 12:28:45.830612 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:45.830578 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46" Oct 30 12:28:45.874641 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:45.874589 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46" podStartSLOduration=1.874576732 podStartE2EDuration="1.874576732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:45.874388728 +0000 UTC m=+2076.908527124" watchObservedRunningTime="2026-10-30 12:28:45.874576732 +0000 UTC m=+2076.908715127" Oct 30 12:28:49.349766 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:49.349731 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh"] Oct 30 12:28:49.352896 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:49.352874 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" Oct 30 12:28:49.391265 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:49.391221 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh"] Oct 30 12:28:49.410371 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:49.410336 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-ab9381-serving-cert\"" Oct 30 12:28:49.541433 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:49.532150 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb116259-4d49-491d-b7e8-09b1c46d9caf-metrics-certs\") pod \"maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh\" (UID: \"fb116259-4d49-491d-b7e8-09b1c46d9caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" Oct 30 12:28:49.541433 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:49.532336 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb116259-4d49-491d-b7e8-09b1c46d9caf-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh\" (UID: \"fb116259-4d49-491d-b7e8-09b1c46d9caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" Oct 30 12:28:49.541433 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:49.532448 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb116259-4d49-491d-b7e8-09b1c46d9caf-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh\" (UID: \"fb116259-4d49-491d-b7e8-09b1c46d9caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" Oct 30 12:28:49.541433 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:49.532484 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnfvp\" (UniqueName: \"kubernetes.io/projected/fb116259-4d49-491d-b7e8-09b1c46d9caf-kube-api-access-rnfvp\") pod \"maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh\" (UID: \"fb116259-4d49-491d-b7e8-09b1c46d9caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" Oct 30 12:28:49.637983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:49.637895 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb116259-4d49-491d-b7e8-09b1c46d9caf-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh\" (UID: \"fb116259-4d49-491d-b7e8-09b1c46d9caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" Oct 30 12:28:49.638234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:49.637999 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb116259-4d49-491d-b7e8-09b1c46d9caf-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh\" (UID: \"fb116259-4d49-491d-b7e8-09b1c46d9caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" Oct 30 12:28:49.638234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:49.638022 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnfvp\" (UniqueName: \"kubernetes.io/projected/fb116259-4d49-491d-b7e8-09b1c46d9caf-kube-api-access-rnfvp\") pod \"maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh\" (UID: \"fb116259-4d49-491d-b7e8-09b1c46d9caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" Oct 30 12:28:49.638234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:49.638109 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb116259-4d49-491d-b7e8-09b1c46d9caf-metrics-certs\") pod \"maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh\" (UID: \"fb116259-4d49-491d-b7e8-09b1c46d9caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" Oct 30 12:28:49.639285 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:49.639257 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb116259-4d49-491d-b7e8-09b1c46d9caf-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh\" (UID: \"fb116259-4d49-491d-b7e8-09b1c46d9caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" Oct 30 12:28:49.641125 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:49.641094 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb116259-4d49-491d-b7e8-09b1c46d9caf-metrics-certs\") pod \"maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh\" (UID: \"fb116259-4d49-491d-b7e8-09b1c46d9caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" Oct 30 12:28:49.642034 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:49.642011 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb116259-4d49-491d-b7e8-09b1c46d9caf-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh\" (UID: \"fb116259-4d49-491d-b7e8-09b1c46d9caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" Oct 30 12:28:49.664575 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:49.664543 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnfvp\" (UniqueName: \"kubernetes.io/projected/fb116259-4d49-491d-b7e8-09b1c46d9caf-kube-api-access-rnfvp\") pod \"maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh\" (UID: \"fb116259-4d49-491d-b7e8-09b1c46d9caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" Oct 30 12:28:49.673108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:49.672265 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" Oct 30 12:28:49.981524 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:28:49.981492 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb116259_4d49_491d_b7e8_09b1c46d9caf.slice/crio-7df6144bdc9f9c619f7597fa2d0ee051c8efe1138f62b43588fccf6b413f30af WatchSource:0}: Error finding container 7df6144bdc9f9c619f7597fa2d0ee051c8efe1138f62b43588fccf6b413f30af: Status 404 returned error can't find the container with id 7df6144bdc9f9c619f7597fa2d0ee051c8efe1138f62b43588fccf6b413f30af Oct 30 12:28:49.984230 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:49.984170 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh"] Oct 30 12:28:50.886591 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:50.886554 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" event={"ID":"fb116259-4d49-491d-b7e8-09b1c46d9caf","Type":"ContainerStarted","Data":"0f110d82ea58326bd85e6c45744c688a82142410b856867f2560c048e7fc0abb"} Oct 30 12:28:50.887120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:50.886606 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" event={"ID":"fb116259-4d49-491d-b7e8-09b1c46d9caf","Type":"ContainerStarted","Data":"7df6144bdc9f9c619f7597fa2d0ee051c8efe1138f62b43588fccf6b413f30af"} Oct 30 12:28:50.887120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:50.886710 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" Oct 30 12:28:50.931896 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:50.931848 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" podStartSLOduration=1.623582253 podStartE2EDuration="1.931834581s" totalImagesPullingTime="308.252328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:50.930921344 +0000 UTC m=+2081.965059741" watchObservedRunningTime="2026-10-30 12:28:50.931834581 +0000 UTC m=+2081.965972977" Oct 30 12:28:53.599580 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:28:53.599542 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:28:56.900686 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:56.900654 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" Oct 30 12:28:57.846269 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:28:57.846232 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46" Oct 30 12:29:11.488144 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:11.487958 4468 patch_prober.go:29] interesting pod/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": read tcp 10.134.0.2:56152->10.134.0.146:15021: read: connection reset by peer" start-of-body= Oct 30 12:29:11.488144 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:11.488031 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" podUID="88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": read tcp 10.134.0.2:56152->10.134.0.146:15021: read: connection reset by peer" Oct 30 12:29:12.217954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:12.217921 4468 generic.go:356] "Generic (PLEG): container finished" podID="88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" containerID="660bc54ecfc8ebbf9214fd099a98bf6167e1e0349d549ca65da2d010150c4b5b" exitCode=0 Oct 30 12:29:12.218251 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:12.218013 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" event={"ID":"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4","Type":"ContainerDied","Data":"660bc54ecfc8ebbf9214fd099a98bf6167e1e0349d549ca65da2d010150c4b5b"} Oct 30 12:29:12.218584 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:12.218562 4468 scope.go:122] "RemoveContainer" containerID="660bc54ecfc8ebbf9214fd099a98bf6167e1e0349d549ca65da2d010150c4b5b" Oct 30 12:29:12.220776 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:12.220740 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:29:12.220897 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:12.220841 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:29:12.220965 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:12.220900 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:29:13.238297 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:13.238251 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" event={"ID":"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4","Type":"ContainerStarted","Data":"191b9aa92fa6fcda99c74542f4deaad0e3c5ab019d41d547b780b8b05f2c5e4a"} Oct 30 12:29:13.637271 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:13.637185 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:29:13.659647 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:13.647274 4468 patch_prober.go:29] interesting pod/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 12:29:13.668913 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:13.660161 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" podUID="88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 12:29:14.625089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:14.618679 4468 patch_prober.go:29] interesting pod/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 12:29:14.625089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:14.618751 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" podUID="88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 12:29:15.619095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:15.614446 4468 patch_prober.go:29] interesting pod/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 12:29:15.619095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:15.614524 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" podUID="88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 12:29:16.618530 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:16.618483 4468 patch_prober.go:29] interesting pod/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 12:29:16.619158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:16.619120 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" podUID="88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 12:29:17.615495 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:17.615453 4468 patch_prober.go:29] interesting pod/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 12:29:17.615825 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:17.615793 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" podUID="88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 12:29:18.702280 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:18.702248 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:29:18.703912 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:18.703873 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:29:18.708100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:18.708047 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:29:18.902501 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:18.902471 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr"] Oct 30 12:29:18.906410 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:18.906387 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" Oct 30 12:29:18.923000 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:18.922972 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c44eee\"" Oct 30 12:29:18.923157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:18.923125 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c44eee-dockercfg-xm48l\"" Oct 30 12:29:18.924498 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:18.924475 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr"] Oct 30 12:29:19.219088 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:19.218340 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b36f2ed4-3517-4110-ab7a-8cfae8243bab-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr\" (UID: \"b36f2ed4-3517-4110-ab7a-8cfae8243bab\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" Oct 30 12:29:19.219088 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:19.218464 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tmjz\" (UniqueName: \"kubernetes.io/projected/b36f2ed4-3517-4110-ab7a-8cfae8243bab-kube-api-access-4tmjz\") pod \"payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr\" (UID: \"b36f2ed4-3517-4110-ab7a-8cfae8243bab\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" Oct 30 12:29:19.319777 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:19.319737 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tmjz\" (UniqueName: \"kubernetes.io/projected/b36f2ed4-3517-4110-ab7a-8cfae8243bab-kube-api-access-4tmjz\") pod \"payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr\" (UID: \"b36f2ed4-3517-4110-ab7a-8cfae8243bab\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" Oct 30 12:29:19.319939 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:19.319930 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b36f2ed4-3517-4110-ab7a-8cfae8243bab-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr\" (UID: \"b36f2ed4-3517-4110-ab7a-8cfae8243bab\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" Oct 30 12:29:19.320794 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:19.320768 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b36f2ed4-3517-4110-ab7a-8cfae8243bab-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr\" (UID: \"b36f2ed4-3517-4110-ab7a-8cfae8243bab\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" Oct 30 12:29:19.347432 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:19.347397 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tmjz\" (UniqueName: \"kubernetes.io/projected/b36f2ed4-3517-4110-ab7a-8cfae8243bab-kube-api-access-4tmjz\") pod \"payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr\" (UID: \"b36f2ed4-3517-4110-ab7a-8cfae8243bab\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" Oct 30 12:29:19.520281 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:19.520205 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" Oct 30 12:29:19.825858 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:29:19.825826 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36f2ed4_3517_4110_ab7a_8cfae8243bab.slice/crio-ba3a17e83d0c19b92f8bea2d3650d37d31ef6cf5c2cce5829f07825ede77ab61 WatchSource:0}: Error finding container ba3a17e83d0c19b92f8bea2d3650d37d31ef6cf5c2cce5829f07825ede77ab61: Status 404 returned error can't find the container with id ba3a17e83d0c19b92f8bea2d3650d37d31ef6cf5c2cce5829f07825ede77ab61 Oct 30 12:29:19.830736 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:19.830705 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr"] Oct 30 12:29:20.285158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:20.285109 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" event={"ID":"b36f2ed4-3517-4110-ab7a-8cfae8243bab","Type":"ContainerStarted","Data":"75a7c0baa2537a337a615134ca9d593d22b8c3ab8f9d6d4acf6cad38922e1a5c"} Oct 30 12:29:20.285351 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:20.285176 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" event={"ID":"b36f2ed4-3517-4110-ab7a-8cfae8243bab","Type":"ContainerStarted","Data":"ba3a17e83d0c19b92f8bea2d3650d37d31ef6cf5c2cce5829f07825ede77ab61"} Oct 30 12:29:20.285415 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:20.285387 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" Oct 30 12:29:20.329614 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:20.329558 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" podStartSLOduration=2.329545848 podStartE2EDuration="2.329545848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:20.327538728 +0000 UTC m=+2111.361677124" watchObservedRunningTime="2026-10-30 12:29:20.329545848 +0000 UTC m=+2111.363684245" Oct 30 12:29:21.420246 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:21.420017 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg"] Oct 30 12:29:21.423961 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:21.423928 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg" Oct 30 12:29:21.448123 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:21.448090 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg"] Oct 30 12:29:21.482209 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:21.482171 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ec236e\"" Oct 30 12:29:21.487965 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:21.487921 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ec236e-dockercfg-gtzwd\"" Oct 30 12:29:21.500948 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:21.499260 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a160a20-d339-40f4-9c7a-8457342868d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg\" (UID: \"8a160a20-d339-40f4-9c7a-8457342868d3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg" Oct 30 12:29:21.500948 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:21.499450 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6tlr\" (UniqueName: \"kubernetes.io/projected/8a160a20-d339-40f4-9c7a-8457342868d3-kube-api-access-z6tlr\") pod \"payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg\" (UID: \"8a160a20-d339-40f4-9c7a-8457342868d3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg" Oct 30 12:29:21.645091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:21.642815 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a160a20-d339-40f4-9c7a-8457342868d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg\" (UID: \"8a160a20-d339-40f4-9c7a-8457342868d3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg" Oct 30 12:29:21.645091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:21.642976 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6tlr\" (UniqueName: \"kubernetes.io/projected/8a160a20-d339-40f4-9c7a-8457342868d3-kube-api-access-z6tlr\") pod \"payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg\" (UID: \"8a160a20-d339-40f4-9c7a-8457342868d3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg" Oct 30 12:29:21.645091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:21.644367 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a160a20-d339-40f4-9c7a-8457342868d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg\" (UID: \"8a160a20-d339-40f4-9c7a-8457342868d3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg" Oct 30 12:29:21.706344 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:21.705826 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6tlr\" (UniqueName: \"kubernetes.io/projected/8a160a20-d339-40f4-9c7a-8457342868d3-kube-api-access-z6tlr\") pod \"payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg\" (UID: \"8a160a20-d339-40f4-9c7a-8457342868d3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg" Oct 30 12:29:21.740226 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:21.740194 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg" Oct 30 12:29:22.145655 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:29:22.145614 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a160a20_d339_40f4_9c7a_8457342868d3.slice/crio-ccfa046c81a6d50d79eaf2b911cd9ea1ee84d5357d327591305c7cfaa1c97e82 WatchSource:0}: Error finding container ccfa046c81a6d50d79eaf2b911cd9ea1ee84d5357d327591305c7cfaa1c97e82: Status 404 returned error can't find the container with id ccfa046c81a6d50d79eaf2b911cd9ea1ee84d5357d327591305c7cfaa1c97e82 Oct 30 12:29:22.148514 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:22.148487 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg"] Oct 30 12:29:22.300964 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:22.300923 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg" event={"ID":"8a160a20-d339-40f4-9c7a-8457342868d3","Type":"ContainerStarted","Data":"4da132724c390b2194583fd1bf72e34ab13cf276364cc7037430678f509ea4a1"} Oct 30 12:29:22.301181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:22.300986 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg" event={"ID":"8a160a20-d339-40f4-9c7a-8457342868d3","Type":"ContainerStarted","Data":"ccfa046c81a6d50d79eaf2b911cd9ea1ee84d5357d327591305c7cfaa1c97e82"} Oct 30 12:29:22.301181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:22.301016 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg" Oct 30 12:29:22.349880 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:22.349826 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg" podStartSLOduration=1.349812107 podStartE2EDuration="1.349812107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:22.344827348 +0000 UTC m=+2113.378965746" watchObservedRunningTime="2026-10-30 12:29:22.349812107 +0000 UTC m=+2113.383950500" Oct 30 12:29:23.657601 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:29:23.657553 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:29:27.579476 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:27.579434 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w"] Oct 30 12:29:27.579929 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:27.579690 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" podUID="189793bd-8385-4741-85aa-3d89fc9a1d30" containerName="istio-proxy" containerID="cri-o://df31c4131c9ad60d5b14a267527ba6c248b524f8001e68c61aec318a0e43e899" gracePeriod=30 Oct 30 12:29:28.225174 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:28.225134 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:29:28.225392 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:28.225199 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" podUID="189793bd-8385-4741-85aa-3d89fc9a1d30" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:29:31.296365 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:31.296329 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" Oct 30 12:29:32.993701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:32.993665 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:29:32.998857 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:32.998821 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-podinfo\") pod \"189793bd-8385-4741-85aa-3d89fc9a1d30\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " Oct 30 12:29:32.999013 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:32.998941 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-token\" (UniqueName: \"kubernetes.io/projected/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-token\") pod \"189793bd-8385-4741-85aa-3d89fc9a1d30\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " Oct 30 12:29:32.999096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:32.999039 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-workload-certs\") pod \"189793bd-8385-4741-85aa-3d89fc9a1d30\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " Oct 30 12:29:32.999218 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:32.999197 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189793bd-8385-4741-85aa-3d89fc9a1d30-kube-api-access-n7p57\" (UniqueName: \"kubernetes.io/projected/189793bd-8385-4741-85aa-3d89fc9a1d30-kube-api-access-n7p57\") pod \"189793bd-8385-4741-85aa-3d89fc9a1d30\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " Oct 30 12:29:32.999340 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:32.999316 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189793bd-8385-4741-85aa-3d89fc9a1d30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189793bd-8385-4741-85aa-3d89fc9a1d30-istiod-ca-cert\") pod \"189793bd-8385-4741-85aa-3d89fc9a1d30\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " Oct 30 12:29:32.999414 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:32.999357 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-workload-certs" pod "189793bd-8385-4741-85aa-3d89fc9a1d30" (UID: "189793bd-8385-4741-85aa-3d89fc9a1d30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:32.999472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:32.999420 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-workload-socket\") pod \"189793bd-8385-4741-85aa-3d89fc9a1d30\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " Oct 30 12:29:32.999529 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:32.999469 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-data\") pod \"189793bd-8385-4741-85aa-3d89fc9a1d30\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " Oct 30 12:29:32.999583 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:32.999531 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-envoy\") pod \"189793bd-8385-4741-85aa-3d89fc9a1d30\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " Oct 30 12:29:32.999583 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:32.999561 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-credential-socket\") pod \"189793bd-8385-4741-85aa-3d89fc9a1d30\" (UID: \"189793bd-8385-4741-85aa-3d89fc9a1d30\") " Oct 30 12:29:32.999807 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:32.999641 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189793bd-8385-4741-85aa-3d89fc9a1d30-istiod-ca-cert" pod "189793bd-8385-4741-85aa-3d89fc9a1d30" (UID: "189793bd-8385-4741-85aa-3d89fc9a1d30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:29:32.999807 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:32.999700 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-workload-socket" pod "189793bd-8385-4741-85aa-3d89fc9a1d30" (UID: "189793bd-8385-4741-85aa-3d89fc9a1d30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:33.000224 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:32.999998 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-data" pod "189793bd-8385-4741-85aa-3d89fc9a1d30" (UID: "189793bd-8385-4741-85aa-3d89fc9a1d30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:33.000299 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.000263 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-credential-socket" pod "189793bd-8385-4741-85aa-3d89fc9a1d30" (UID: "189793bd-8385-4741-85aa-3d89fc9a1d30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:33.000498 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.000478 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189793bd-8385-4741-85aa-3d89fc9a1d30-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:29:33.000585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.000503 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:29:33.000585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.000516 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:29:33.000585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.000530 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:29:33.000585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.000545 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:29:33.002406 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.002371 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-podinfo" pod "189793bd-8385-4741-85aa-3d89fc9a1d30" (UID: "189793bd-8385-4741-85aa-3d89fc9a1d30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:29:33.002512 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.002492 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-token" pod "189793bd-8385-4741-85aa-3d89fc9a1d30" (UID: "189793bd-8385-4741-85aa-3d89fc9a1d30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:33.002576 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.002562 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-envoy" pod "189793bd-8385-4741-85aa-3d89fc9a1d30" (UID: "189793bd-8385-4741-85aa-3d89fc9a1d30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:33.002663 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.002643 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189793bd-8385-4741-85aa-3d89fc9a1d30-kube-api-access-n7p57" pod "189793bd-8385-4741-85aa-3d89fc9a1d30" (UID: "189793bd-8385-4741-85aa-3d89fc9a1d30"). InnerVolumeSpecName "kube-api-access-n7p57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:33.102283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.102246 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:29:33.102283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.102278 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:29:33.102283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.102288 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189793bd-8385-4741-85aa-3d89fc9a1d30-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:29:33.102612 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.102295 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7p57\" (UniqueName: \"kubernetes.io/projected/189793bd-8385-4741-85aa-3d89fc9a1d30-kube-api-access-n7p57\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:29:33.330461 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.330354 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg" Oct 30 12:29:33.462756 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.462704 4468 generic.go:356] "Generic (PLEG): container finished" podID="189793bd-8385-4741-85aa-3d89fc9a1d30" containerID="df31c4131c9ad60d5b14a267527ba6c248b524f8001e68c61aec318a0e43e899" exitCode=0 Oct 30 12:29:33.462961 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.462757 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" Oct 30 12:29:33.462961 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.462794 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" event={"ID":"189793bd-8385-4741-85aa-3d89fc9a1d30","Type":"ContainerDied","Data":"df31c4131c9ad60d5b14a267527ba6c248b524f8001e68c61aec318a0e43e899"} Oct 30 12:29:33.462961 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.462863 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w" event={"ID":"189793bd-8385-4741-85aa-3d89fc9a1d30","Type":"ContainerDied","Data":"7b8d1aabfd2b83485adfe2fd5b34676cad3cbda7c3c97e12b7fe601ccc68fad8"} Oct 30 12:29:33.462961 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.462895 4468 scope.go:122] "RemoveContainer" containerID="df31c4131c9ad60d5b14a267527ba6c248b524f8001e68c61aec318a0e43e899" Oct 30 12:29:33.475317 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.475291 4468 scope.go:122] "RemoveContainer" containerID="8bc6e9749f1b5d86d39134cea2c68e807cab5bcd4fbfeecd1d587e2919976070" Oct 30 12:29:33.496852 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.496829 4468 scope.go:122] "RemoveContainer" containerID="df31c4131c9ad60d5b14a267527ba6c248b524f8001e68c61aec318a0e43e899" Oct 30 12:29:33.497249 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:29:33.497225 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df31c4131c9ad60d5b14a267527ba6c248b524f8001e68c61aec318a0e43e899\": container with ID starting with df31c4131c9ad60d5b14a267527ba6c248b524f8001e68c61aec318a0e43e899 not found: ID does not exist" containerID="df31c4131c9ad60d5b14a267527ba6c248b524f8001e68c61aec318a0e43e899" Oct 30 12:29:33.497339 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.497256 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df31c4131c9ad60d5b14a267527ba6c248b524f8001e68c61aec318a0e43e899"} err="failed to get container status \"df31c4131c9ad60d5b14a267527ba6c248b524f8001e68c61aec318a0e43e899\": rpc error: code = NotFound desc = could not find container \"df31c4131c9ad60d5b14a267527ba6c248b524f8001e68c61aec318a0e43e899\": container with ID starting with df31c4131c9ad60d5b14a267527ba6c248b524f8001e68c61aec318a0e43e899 not found: ID does not exist" Oct 30 12:29:33.497339 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.497275 4468 scope.go:122] "RemoveContainer" containerID="8bc6e9749f1b5d86d39134cea2c68e807cab5bcd4fbfeecd1d587e2919976070" Oct 30 12:29:33.497557 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:29:33.497535 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc6e9749f1b5d86d39134cea2c68e807cab5bcd4fbfeecd1d587e2919976070\": container with ID starting with 8bc6e9749f1b5d86d39134cea2c68e807cab5bcd4fbfeecd1d587e2919976070 not found: ID does not exist" containerID="8bc6e9749f1b5d86d39134cea2c68e807cab5bcd4fbfeecd1d587e2919976070" Oct 30 12:29:33.497611 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.497564 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc6e9749f1b5d86d39134cea2c68e807cab5bcd4fbfeecd1d587e2919976070"} err="failed to get container status \"8bc6e9749f1b5d86d39134cea2c68e807cab5bcd4fbfeecd1d587e2919976070\": rpc error: code = NotFound desc = could not find container \"8bc6e9749f1b5d86d39134cea2c68e807cab5bcd4fbfeecd1d587e2919976070\": container with ID starting with 8bc6e9749f1b5d86d39134cea2c68e807cab5bcd4fbfeecd1d587e2919976070 not found: ID does not exist" Oct 30 12:29:33.535035 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.535000 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w"] Oct 30 12:29:33.540513 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:33.540484 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-wrk6w"] Oct 30 12:29:35.095052 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:35.094974 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189793bd-8385-4741-85aa-3d89fc9a1d30" path="/var/lib/kubelet/pods/189793bd-8385-4741-85aa-3d89fc9a1d30/volumes" Oct 30 12:29:35.962046 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:35.962007 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz"] Oct 30 12:29:35.962332 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:35.962277 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" podUID="468d055f-47b8-4481-84cb-80afefd5ace3" containerName="istio-proxy" containerID="cri-o://2d29b02e1ef908a8340c3fc375111b40b7b773ba81f94ba3e5548413854d03e1" gracePeriod=30 Oct 30 12:29:40.426262 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.426224 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7"] Oct 30 12:29:40.426939 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.426925 4468 cpu_manager.go:414] "containerMap: removing container" podUID="189793bd-8385-4741-85aa-3d89fc9a1d30" containerName="istio-proxy" Oct 30 12:29:40.426939 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.426941 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189793bd-8385-4741-85aa-3d89fc9a1d30" containerName="istio-proxy" Oct 30 12:29:40.427027 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.426981 4468 cpu_manager.go:414] "containerMap: removing container" podUID="189793bd-8385-4741-85aa-3d89fc9a1d30" containerName="istio-proxy" Oct 30 12:29:40.427027 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.426988 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189793bd-8385-4741-85aa-3d89fc9a1d30" containerName="istio-proxy" Oct 30 12:29:40.427144 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.427133 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="189793bd-8385-4741-85aa-3d89fc9a1d30" containerName="istio-proxy" Oct 30 12:29:40.430593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.430568 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" Oct 30 12:29:40.450564 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.450526 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-c44eee-serving-cert\"" Oct 30 12:29:40.468138 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.455609 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7"] Oct 30 12:29:40.508064 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.508022 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnfs5\" (UniqueName: \"kubernetes.io/projected/04fb37c3-e5de-408c-bb27-88502637597e-kube-api-access-dnfs5\") pod \"maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7\" (UID: \"04fb37c3-e5de-408c-bb27-88502637597e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" Oct 30 12:29:40.508277 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.508183 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04fb37c3-e5de-408c-bb27-88502637597e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7\" (UID: \"04fb37c3-e5de-408c-bb27-88502637597e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" Oct 30 12:29:40.508411 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.508388 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04fb37c3-e5de-408c-bb27-88502637597e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7\" (UID: \"04fb37c3-e5de-408c-bb27-88502637597e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" Oct 30 12:29:40.508484 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.508452 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04fb37c3-e5de-408c-bb27-88502637597e-metrics-certs\") pod \"maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7\" (UID: \"04fb37c3-e5de-408c-bb27-88502637597e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" Oct 30 12:29:40.624812 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.619342 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04fb37c3-e5de-408c-bb27-88502637597e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7\" (UID: \"04fb37c3-e5de-408c-bb27-88502637597e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" Oct 30 12:29:40.624812 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.619524 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04fb37c3-e5de-408c-bb27-88502637597e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7\" (UID: \"04fb37c3-e5de-408c-bb27-88502637597e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" Oct 30 12:29:40.624812 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.619585 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04fb37c3-e5de-408c-bb27-88502637597e-metrics-certs\") pod \"maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7\" (UID: \"04fb37c3-e5de-408c-bb27-88502637597e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" Oct 30 12:29:40.624812 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.619778 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnfs5\" (UniqueName: \"kubernetes.io/projected/04fb37c3-e5de-408c-bb27-88502637597e-kube-api-access-dnfs5\") pod \"maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7\" (UID: \"04fb37c3-e5de-408c-bb27-88502637597e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" Oct 30 12:29:40.624812 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.621141 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04fb37c3-e5de-408c-bb27-88502637597e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7\" (UID: \"04fb37c3-e5de-408c-bb27-88502637597e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" Oct 30 12:29:40.669095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.661864 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04fb37c3-e5de-408c-bb27-88502637597e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7\" (UID: \"04fb37c3-e5de-408c-bb27-88502637597e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" Oct 30 12:29:40.680210 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.679898 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnfs5\" (UniqueName: \"kubernetes.io/projected/04fb37c3-e5de-408c-bb27-88502637597e-kube-api-access-dnfs5\") pod \"maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7\" (UID: \"04fb37c3-e5de-408c-bb27-88502637597e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" Oct 30 12:29:40.681586 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.681450 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04fb37c3-e5de-408c-bb27-88502637597e-metrics-certs\") pod \"maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7\" (UID: \"04fb37c3-e5de-408c-bb27-88502637597e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" Oct 30 12:29:40.751524 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:40.751488 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" Oct 30 12:29:41.067733 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:29:41.067687 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04fb37c3_e5de_408c_bb27_88502637597e.slice/crio-74a6c56fe0ae6632d2ed7e844c7e27297ac7760e30258358ba4f2b4aa124faca WatchSource:0}: Error finding container 74a6c56fe0ae6632d2ed7e844c7e27297ac7760e30258358ba4f2b4aa124faca: Status 404 returned error can't find the container with id 74a6c56fe0ae6632d2ed7e844c7e27297ac7760e30258358ba4f2b4aa124faca Oct 30 12:29:41.073815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.073745 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7"] Oct 30 12:29:41.163906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.163877 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:29:41.262803 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.262244 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468d055f-47b8-4481-84cb-80afefd5ace3-istio-token\" (UniqueName: \"kubernetes.io/projected/468d055f-47b8-4481-84cb-80afefd5ace3-istio-token\") pod \"468d055f-47b8-4481-84cb-80afefd5ace3\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " Oct 30 12:29:41.263539 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.263510 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/468d055f-47b8-4481-84cb-80afefd5ace3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468d055f-47b8-4481-84cb-80afefd5ace3-istio-podinfo\") pod \"468d055f-47b8-4481-84cb-80afefd5ace3\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " Oct 30 12:29:41.263797 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.263777 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-workload-socket\") pod \"468d055f-47b8-4481-84cb-80afefd5ace3\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " Oct 30 12:29:41.264059 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.264040 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-istio-data\") pod \"468d055f-47b8-4481-84cb-80afefd5ace3\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " Oct 30 12:29:41.264431 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.264397 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468d055f-47b8-4481-84cb-80afefd5ace3-kube-api-access-68j2w\" (UniqueName: \"kubernetes.io/projected/468d055f-47b8-4481-84cb-80afefd5ace3-kube-api-access-68j2w\") pod \"468d055f-47b8-4481-84cb-80afefd5ace3\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " Oct 30 12:29:41.265158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.265001 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-istio-envoy\") pod \"468d055f-47b8-4481-84cb-80afefd5ace3\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " Oct 30 12:29:41.265158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.265049 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-credential-socket\") pod \"468d055f-47b8-4481-84cb-80afefd5ace3\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " Oct 30 12:29:41.265158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.264565 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-workload-socket" pod "468d055f-47b8-4481-84cb-80afefd5ace3" (UID: "468d055f-47b8-4481-84cb-80afefd5ace3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:41.265158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.264819 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-istio-data" pod "468d055f-47b8-4481-84cb-80afefd5ace3" (UID: "468d055f-47b8-4481-84cb-80afefd5ace3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:41.266238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.265213 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-workload-certs\") pod \"468d055f-47b8-4481-84cb-80afefd5ace3\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " Oct 30 12:29:41.266238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.265382 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468d055f-47b8-4481-84cb-80afefd5ace3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468d055f-47b8-4481-84cb-80afefd5ace3-istiod-ca-cert\") pod \"468d055f-47b8-4481-84cb-80afefd5ace3\" (UID: \"468d055f-47b8-4481-84cb-80afefd5ace3\") " Oct 30 12:29:41.266238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.265553 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-credential-socket" pod "468d055f-47b8-4481-84cb-80afefd5ace3" (UID: "468d055f-47b8-4481-84cb-80afefd5ace3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:41.266238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.265624 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-workload-certs" pod "468d055f-47b8-4481-84cb-80afefd5ace3" (UID: "468d055f-47b8-4481-84cb-80afefd5ace3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:41.266238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.265976 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468d055f-47b8-4481-84cb-80afefd5ace3-istiod-ca-cert" pod "468d055f-47b8-4481-84cb-80afefd5ace3" (UID: "468d055f-47b8-4481-84cb-80afefd5ace3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:29:41.266238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.266208 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:29:41.266238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.266233 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468d055f-47b8-4481-84cb-80afefd5ace3-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:29:41.266672 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.266246 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:29:41.266672 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.266259 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:29:41.266672 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.266273 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:29:41.267563 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.267536 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468d055f-47b8-4481-84cb-80afefd5ace3-istio-token" pod "468d055f-47b8-4481-84cb-80afefd5ace3" (UID: "468d055f-47b8-4481-84cb-80afefd5ace3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:41.267807 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.267776 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468d055f-47b8-4481-84cb-80afefd5ace3-kube-api-access-68j2w" pod "468d055f-47b8-4481-84cb-80afefd5ace3" (UID: "468d055f-47b8-4481-84cb-80afefd5ace3"). InnerVolumeSpecName "kube-api-access-68j2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:41.268015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.267988 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-istio-envoy" pod "468d055f-47b8-4481-84cb-80afefd5ace3" (UID: "468d055f-47b8-4481-84cb-80afefd5ace3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:41.268566 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.268542 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/468d055f-47b8-4481-84cb-80afefd5ace3-istio-podinfo" pod "468d055f-47b8-4481-84cb-80afefd5ace3" (UID: "468d055f-47b8-4481-84cb-80afefd5ace3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:29:41.367818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.367775 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468d055f-47b8-4481-84cb-80afefd5ace3-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:29:41.367818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.367812 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468d055f-47b8-4481-84cb-80afefd5ace3-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:29:41.367818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.367822 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68j2w\" (UniqueName: \"kubernetes.io/projected/468d055f-47b8-4481-84cb-80afefd5ace3-kube-api-access-68j2w\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:29:41.367818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.367832 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468d055f-47b8-4481-84cb-80afefd5ace3-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:29:41.628162 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.625543 4468 generic.go:356] "Generic (PLEG): container finished" podID="468d055f-47b8-4481-84cb-80afefd5ace3" containerID="2d29b02e1ef908a8340c3fc375111b40b7b773ba81f94ba3e5548413854d03e1" exitCode=0 Oct 30 12:29:41.628162 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.625753 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" event={"ID":"468d055f-47b8-4481-84cb-80afefd5ace3","Type":"ContainerDied","Data":"2d29b02e1ef908a8340c3fc375111b40b7b773ba81f94ba3e5548413854d03e1"} Oct 30 12:29:41.628162 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.625811 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" event={"ID":"468d055f-47b8-4481-84cb-80afefd5ace3","Type":"ContainerDied","Data":"3f4bcdf341890ede44e78a8d380e46699b181f3a1db7fa616d0e31d4db02e0d2"} Oct 30 12:29:41.628162 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.625843 4468 scope.go:122] "RemoveContainer" containerID="2d29b02e1ef908a8340c3fc375111b40b7b773ba81f94ba3e5548413854d03e1" Oct 30 12:29:41.628162 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.626009 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz" Oct 30 12:29:41.659801 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.639794 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" event={"ID":"04fb37c3-e5de-408c-bb27-88502637597e","Type":"ContainerStarted","Data":"d30421e56879d6062631d7d4b42cda0e9fbdbac025c7b4ad66cd99af37bc7f82"} Oct 30 12:29:41.659801 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.641032 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" event={"ID":"04fb37c3-e5de-408c-bb27-88502637597e","Type":"ContainerStarted","Data":"74a6c56fe0ae6632d2ed7e844c7e27297ac7760e30258358ba4f2b4aa124faca"} Oct 30 12:29:41.659801 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.641163 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" Oct 30 12:29:41.703602 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.703475 4468 scope.go:122] "RemoveContainer" containerID="2d29b02e1ef908a8340c3fc375111b40b7b773ba81f94ba3e5548413854d03e1" Oct 30 12:29:41.703893 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:29:41.703860 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d29b02e1ef908a8340c3fc375111b40b7b773ba81f94ba3e5548413854d03e1\": container with ID starting with 2d29b02e1ef908a8340c3fc375111b40b7b773ba81f94ba3e5548413854d03e1 not found: ID does not exist" containerID="2d29b02e1ef908a8340c3fc375111b40b7b773ba81f94ba3e5548413854d03e1" Oct 30 12:29:41.704001 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.703901 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d29b02e1ef908a8340c3fc375111b40b7b773ba81f94ba3e5548413854d03e1"} err="failed to get container status \"2d29b02e1ef908a8340c3fc375111b40b7b773ba81f94ba3e5548413854d03e1\": rpc error: code = NotFound desc = could not find container \"2d29b02e1ef908a8340c3fc375111b40b7b773ba81f94ba3e5548413854d03e1\": container with ID starting with 2d29b02e1ef908a8340c3fc375111b40b7b773ba81f94ba3e5548413854d03e1 not found: ID does not exist" Oct 30 12:29:41.725837 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.725783 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" podStartSLOduration=1.407980928 podStartE2EDuration="1.725766425s" totalImagesPullingTime="317.785497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:41.723386349 +0000 UTC m=+2132.757524745" watchObservedRunningTime="2026-10-30 12:29:41.725766425 +0000 UTC m=+2132.759904821" Oct 30 12:29:41.766317 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.766271 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz"] Oct 30 12:29:41.771611 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:41.771571 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-t88qz"] Oct 30 12:29:43.090488 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:43.090408 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="468d055f-47b8-4481-84cb-80afefd5ace3" path="/var/lib/kubelet/pods/468d055f-47b8-4481-84cb-80afefd5ace3/volumes" Oct 30 12:29:43.657528 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:43.657499 4468 generic.go:356] "Generic (PLEG): container finished" podID="0166c51a-8d66-47e5-ba9a-2fb14b342215" containerID="63d2a29bfbed3a8d0e2aad6a1b5d57f37cf82702804e1b3638fda411f6f17eb4" exitCode=0 Oct 30 12:29:43.657710 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:43.657574 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" event={"ID":"0166c51a-8d66-47e5-ba9a-2fb14b342215","Type":"ContainerDied","Data":"63d2a29bfbed3a8d0e2aad6a1b5d57f37cf82702804e1b3638fda411f6f17eb4"} Oct 30 12:29:43.658044 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:43.658023 4468 scope.go:122] "RemoveContainer" containerID="63d2a29bfbed3a8d0e2aad6a1b5d57f37cf82702804e1b3638fda411f6f17eb4" Oct 30 12:29:43.659642 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:43.659611 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:29:43.659743 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:43.659679 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:29:43.659743 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:43.659725 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:29:44.669176 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.669131 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" event={"ID":"0166c51a-8d66-47e5-ba9a-2fb14b342215","Type":"ContainerStarted","Data":"e59b16c0ba13ce257c13c780a8c300f573a6b83394a53273661b52c441d8f0c3"} Oct 30 12:29:44.774918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.774882 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p"] Oct 30 12:29:44.775609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.775588 4468 cpu_manager.go:414] "containerMap: removing container" podUID="468d055f-47b8-4481-84cb-80afefd5ace3" containerName="istio-proxy" Oct 30 12:29:44.775609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.775609 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468d055f-47b8-4481-84cb-80afefd5ace3" containerName="istio-proxy" Oct 30 12:29:44.775825 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.775811 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="189793bd-8385-4741-85aa-3d89fc9a1d30" containerName="istio-proxy" Oct 30 12:29:44.775903 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.775836 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="468d055f-47b8-4481-84cb-80afefd5ace3" containerName="istio-proxy" Oct 30 12:29:44.778291 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.778271 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" Oct 30 12:29:44.798651 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.798624 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-ec236e-serving-cert\"" Oct 30 12:29:44.799946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.799920 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p"] Oct 30 12:29:44.800756 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.800737 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:29:44.861626 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.861595 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrmnd\" (UniqueName: \"kubernetes.io/projected/f1d20bcb-72ac-4692-9f8e-a064a2886bce-kube-api-access-zrmnd\") pod \"maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p\" (UID: \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" Oct 30 12:29:44.861831 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.861741 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d20bcb-72ac-4692-9f8e-a064a2886bce-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p\" (UID: \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" Oct 30 12:29:44.861894 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.861866 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d20bcb-72ac-4692-9f8e-a064a2886bce-metrics-certs\") pod \"maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p\" (UID: \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" Oct 30 12:29:44.861894 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.861886 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d20bcb-72ac-4692-9f8e-a064a2886bce-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p\" (UID: \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" Oct 30 12:29:44.963840 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.963758 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrmnd\" (UniqueName: \"kubernetes.io/projected/f1d20bcb-72ac-4692-9f8e-a064a2886bce-kube-api-access-zrmnd\") pod \"maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p\" (UID: \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" Oct 30 12:29:44.963995 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.963871 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d20bcb-72ac-4692-9f8e-a064a2886bce-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p\" (UID: \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" Oct 30 12:29:44.963995 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.963964 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d20bcb-72ac-4692-9f8e-a064a2886bce-metrics-certs\") pod \"maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p\" (UID: \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" Oct 30 12:29:44.963995 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.963983 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d20bcb-72ac-4692-9f8e-a064a2886bce-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p\" (UID: \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" Oct 30 12:29:44.964211 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:29:44.964158 4468 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-serving-cert: secret "maas-api-e2e-praxis-b-ec236e-serving-cert" not found Oct 30 12:29:44.964253 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:29:44.964246 4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d20bcb-72ac-4692-9f8e-a064a2886bce-maas-api-tls podName:f1d20bcb-72ac-4692-9f8e-a064a2886bce nodeName:}" failed. No retries permitted until 2026-10-30 12:29:45.464226907 +0000 UTC m=+2136.498365282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f1d20bcb-72ac-4692-9f8e-a064a2886bce-maas-api-tls") pod "maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" (UID: "f1d20bcb-72ac-4692-9f8e-a064a2886bce") : secret "maas-api-e2e-praxis-b-ec236e-serving-cert" not found Oct 30 12:29:44.964594 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.964569 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d20bcb-72ac-4692-9f8e-a064a2886bce-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p\" (UID: \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" Oct 30 12:29:44.966472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.966449 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d20bcb-72ac-4692-9f8e-a064a2886bce-metrics-certs\") pod \"maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p\" (UID: \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" Oct 30 12:29:44.989285 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:44.989254 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrmnd\" (UniqueName: \"kubernetes.io/projected/f1d20bcb-72ac-4692-9f8e-a064a2886bce-kube-api-access-zrmnd\") pod \"maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p\" (UID: \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" Oct 30 12:29:45.473343 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:45.473296 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d20bcb-72ac-4692-9f8e-a064a2886bce-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p\" (UID: \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" Oct 30 12:29:45.476058 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:45.476037 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d20bcb-72ac-4692-9f8e-a064a2886bce-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p\" (UID: \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" Oct 30 12:29:45.481913 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:45.481892 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:29:45.483388 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:45.483367 4468 patch_prober.go:29] interesting pod/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 12:29:45.483471 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:45.483415 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" podUID="0166c51a-8d66-47e5-ba9a-2fb14b342215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 12:29:45.691167 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:45.691134 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" Oct 30 12:29:46.070457 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:29:46.070423 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d20bcb_72ac_4692_9f8e_a064a2886bce.slice/crio-ad3ccce2b2485bd62e77b04cd0227a57cf7ea638fe88fc445418dc2f1a260508 WatchSource:0}: Error finding container ad3ccce2b2485bd62e77b04cd0227a57cf7ea638fe88fc445418dc2f1a260508: Status 404 returned error can't find the container with id ad3ccce2b2485bd62e77b04cd0227a57cf7ea638fe88fc445418dc2f1a260508 Oct 30 12:29:46.078873 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:46.077364 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p"] Oct 30 12:29:46.088220 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:46.088192 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp"] Oct 30 12:29:46.091990 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:46.091973 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp" Oct 30 12:29:46.133768 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:46.133736 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-1e9675-dockercfg-2qvxs\"" Oct 30 12:29:46.136417 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:46.136382 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-1e9675\"" Oct 30 12:29:46.139687 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:46.139652 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp"] Oct 30 12:29:46.194389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:46.194345 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfd7g\" (UniqueName: \"kubernetes.io/projected/42e6150b-6f32-4d92-911c-962f73ba0e68-kube-api-access-jfd7g\") pod \"payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp\" (UID: \"42e6150b-6f32-4d92-911c-962f73ba0e68\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp" Oct 30 12:29:46.194672 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:46.194485 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e6150b-6f32-4d92-911c-962f73ba0e68-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp\" (UID: \"42e6150b-6f32-4d92-911c-962f73ba0e68\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp" Oct 30 12:29:46.308152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:46.308056 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e6150b-6f32-4d92-911c-962f73ba0e68-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp\" (UID: \"42e6150b-6f32-4d92-911c-962f73ba0e68\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp" Oct 30 12:29:46.363149 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:46.308708 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfd7g\" (UniqueName: \"kubernetes.io/projected/42e6150b-6f32-4d92-911c-962f73ba0e68-kube-api-access-jfd7g\") pod \"payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp\" (UID: \"42e6150b-6f32-4d92-911c-962f73ba0e68\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp" Oct 30 12:29:46.377685 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:46.337235 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e6150b-6f32-4d92-911c-962f73ba0e68-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp\" (UID: \"42e6150b-6f32-4d92-911c-962f73ba0e68\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp" Oct 30 12:29:46.473119 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:46.472968 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfd7g\" (UniqueName: \"kubernetes.io/projected/42e6150b-6f32-4d92-911c-962f73ba0e68-kube-api-access-jfd7g\") pod \"payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp\" (UID: \"42e6150b-6f32-4d92-911c-962f73ba0e68\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp" Oct 30 12:29:46.481950 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:46.481920 4468 patch_prober.go:29] interesting pod/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 12:29:46.482145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:46.481967 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" podUID="0166c51a-8d66-47e5-ba9a-2fb14b342215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 12:29:46.744766 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:46.744726 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp" Oct 30 12:29:46.750692 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:46.750663 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" event={"ID":"f1d20bcb-72ac-4692-9f8e-a064a2886bce","Type":"ContainerStarted","Data":"5664ccc038772a6c801f14306556ef2b7f64a37585a0f13633fd5b69b5b43445"} Oct 30 12:29:46.751486 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:46.751469 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" Oct 30 12:29:46.751618 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:46.751605 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" event={"ID":"f1d20bcb-72ac-4692-9f8e-a064a2886bce","Type":"ContainerStarted","Data":"ad3ccce2b2485bd62e77b04cd0227a57cf7ea638fe88fc445418dc2f1a260508"} Oct 30 12:29:46.852099 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:46.849413 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" podStartSLOduration=2.3819298030000002 podStartE2EDuration="2.84939112s" totalImagesPullingTime="467.461317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:46.806880337 +0000 UTC m=+2137.841018740" watchObservedRunningTime="2026-10-30 12:29:46.84939112 +0000 UTC m=+2137.883529517" Oct 30 12:29:47.450953 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:47.450917 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp"] Oct 30 12:29:47.482261 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:47.482234 4468 patch_prober.go:29] interesting pod/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 12:29:47.482375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:47.482287 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" podUID="0166c51a-8d66-47e5-ba9a-2fb14b342215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 12:29:47.680500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:47.680469 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" Oct 30 12:29:47.789524 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:47.789300 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp" event={"ID":"42e6150b-6f32-4d92-911c-962f73ba0e68","Type":"ContainerStarted","Data":"652eaa215dc8f821031626c40a207b5aa4077ef5e7e5771d0cbaf512db68e65a"} Oct 30 12:29:47.789524 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:47.789361 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp" event={"ID":"42e6150b-6f32-4d92-911c-962f73ba0e68","Type":"ContainerStarted","Data":"9129e880cadfe031fdcede0ab0ab82182094468dd3348a14f3bcd4a1f003a1ef"} Oct 30 12:29:47.789524 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:47.789420 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp" Oct 30 12:29:47.848291 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:47.848228 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp" podStartSLOduration=1.8482094359999999 podStartE2EDuration="1.848209436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:47.836599603 +0000 UTC m=+2138.870738000" watchObservedRunningTime="2026-10-30 12:29:47.848209436 +0000 UTC m=+2138.882347836" Oct 30 12:29:48.481876 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:48.481842 4468 patch_prober.go:29] interesting pod/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 12:29:48.482134 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:48.481895 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" podUID="0166c51a-8d66-47e5-ba9a-2fb14b342215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 12:29:49.481881 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:49.481841 4468 patch_prober.go:29] interesting pod/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 12:29:49.482359 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:49.481893 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" podUID="0166c51a-8d66-47e5-ba9a-2fb14b342215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 12:29:50.487224 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:50.487192 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:29:50.488393 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:50.488372 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:29:53.303238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:53.303207 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r_7dc9eb17-6f1d-4246-9849-9a0d75022e41/payload-processing/0.log" Oct 30 12:29:53.547032 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:53.547005 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-ftmv4_910664f0-5127-40e1-b2c8-493f050457cc/payload-processing/0.log" Oct 30 12:29:53.599841 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:29:53.599754 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:29:53.776726 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:53.776698 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-ftmv4_910664f0-5127-40e1-b2c8-493f050457cc/payload-processing/0.log" Oct 30 12:29:53.800400 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:53.800372 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" Oct 30 12:29:53.966664 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:53.966631 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22_b6055421-f8de-4e98-bd0e-4f1a0cbeefc5/payload-processing/0.log" Oct 30 12:29:54.175466 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:54.175434 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22_b6055421-f8de-4e98-bd0e-4f1a0cbeefc5/payload-processing/0.log" Oct 30 12:29:59.804760 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:29:59.804732 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp" Oct 30 12:30:00.208276 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.208238 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2"] Oct 30 12:30:00.213282 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.213251 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" Oct 30 12:30:00.214946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.214916 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg"] Oct 30 12:30:00.314332 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.302906 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:30:00.356311 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.341904 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d790e0f0-b3bc-4a27-aaaf-605ddd0e4595-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2\" (UID: \"d790e0f0-b3bc-4a27-aaaf-605ddd0e4595\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" Oct 30 12:30:00.356624 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.356596 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px9q8\" (UniqueName: \"kubernetes.io/projected/d790e0f0-b3bc-4a27-aaaf-605ddd0e4595-kube-api-access-px9q8\") pod \"maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2\" (UID: \"d790e0f0-b3bc-4a27-aaaf-605ddd0e4595\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" Oct 30 12:30:00.389727 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.380200 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2"] Oct 30 12:30:00.389727 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.380250 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg"] Oct 30 12:30:00.389727 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.380353 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" Oct 30 12:30:00.394233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.394199 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v"] Oct 30 12:30:00.407193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.407146 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" Oct 30 12:30:00.428179 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.425239 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v"] Oct 30 12:30:00.457761 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.457731 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-957nw\" (UniqueName: \"kubernetes.io/projected/3ddcfacd-6aa3-41d2-957f-21919ee15afd-kube-api-access-957nw\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg\" (UID: \"3ddcfacd-6aa3-41d2-957f-21919ee15afd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" Oct 30 12:30:00.457902 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.457853 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d790e0f0-b3bc-4a27-aaaf-605ddd0e4595-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2\" (UID: \"d790e0f0-b3bc-4a27-aaaf-605ddd0e4595\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" Oct 30 12:30:00.457902 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.457885 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px9q8\" (UniqueName: \"kubernetes.io/projected/d790e0f0-b3bc-4a27-aaaf-605ddd0e4595-kube-api-access-px9q8\") pod \"maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2\" (UID: \"d790e0f0-b3bc-4a27-aaaf-605ddd0e4595\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" Oct 30 12:30:00.458008 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.457994 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ddcfacd-6aa3-41d2-957f-21919ee15afd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg\" (UID: \"3ddcfacd-6aa3-41d2-957f-21919ee15afd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" Oct 30 12:30:00.458625 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.458587 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d790e0f0-b3bc-4a27-aaaf-605ddd0e4595-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2\" (UID: \"d790e0f0-b3bc-4a27-aaaf-605ddd0e4595\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" Oct 30 12:30:00.483332 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.483298 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px9q8\" (UniqueName: \"kubernetes.io/projected/d790e0f0-b3bc-4a27-aaaf-605ddd0e4595-kube-api-access-px9q8\") pod \"maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2\" (UID: \"d790e0f0-b3bc-4a27-aaaf-605ddd0e4595\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" Oct 30 12:30:00.559689 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.559655 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ddcfacd-6aa3-41d2-957f-21919ee15afd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg\" (UID: \"3ddcfacd-6aa3-41d2-957f-21919ee15afd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" Oct 30 12:30:00.559877 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.559715 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-957nw\" (UniqueName: \"kubernetes.io/projected/3ddcfacd-6aa3-41d2-957f-21919ee15afd-kube-api-access-957nw\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg\" (UID: \"3ddcfacd-6aa3-41d2-957f-21919ee15afd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" Oct 30 12:30:00.559877 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.559774 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957c1c16-b7c0-4743-927f-0152ac003fcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v\" (UID: \"957c1c16-b7c0-4743-927f-0152ac003fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" Oct 30 12:30:00.559999 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.559963 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqgk2\" (UniqueName: \"kubernetes.io/projected/957c1c16-b7c0-4743-927f-0152ac003fcd-kube-api-access-dqgk2\") pod \"maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v\" (UID: \"957c1c16-b7c0-4743-927f-0152ac003fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" Oct 30 12:30:00.560368 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.560349 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ddcfacd-6aa3-41d2-957f-21919ee15afd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg\" (UID: \"3ddcfacd-6aa3-41d2-957f-21919ee15afd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" Oct 30 12:30:00.584756 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.584720 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-957nw\" (UniqueName: \"kubernetes.io/projected/3ddcfacd-6aa3-41d2-957f-21919ee15afd-kube-api-access-957nw\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg\" (UID: \"3ddcfacd-6aa3-41d2-957f-21919ee15afd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" Oct 30 12:30:00.641234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.641196 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" Oct 30 12:30:00.661376 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.661347 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957c1c16-b7c0-4743-927f-0152ac003fcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v\" (UID: \"957c1c16-b7c0-4743-927f-0152ac003fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" Oct 30 12:30:00.661549 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.661526 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqgk2\" (UniqueName: \"kubernetes.io/projected/957c1c16-b7c0-4743-927f-0152ac003fcd-kube-api-access-dqgk2\") pod \"maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v\" (UID: \"957c1c16-b7c0-4743-927f-0152ac003fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" Oct 30 12:30:00.662183 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.662157 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957c1c16-b7c0-4743-927f-0152ac003fcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v\" (UID: \"957c1c16-b7c0-4743-927f-0152ac003fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" Oct 30 12:30:00.688092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.687107 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqgk2\" (UniqueName: \"kubernetes.io/projected/957c1c16-b7c0-4743-927f-0152ac003fcd-kube-api-access-dqgk2\") pod \"maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v\" (UID: \"957c1c16-b7c0-4743-927f-0152ac003fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" Oct 30 12:30:00.752095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.750587 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" Oct 30 12:30:00.757362 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:00.757104 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" Oct 30 12:30:01.066312 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:01.066272 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2"] Oct 30 12:30:01.275976 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:30:01.275947 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ddcfacd_6aa3_41d2_957f_21919ee15afd.slice/crio-d4754b6cec0099461228feaff2e73b5e052959631194b6880a7019cb08d57c62 WatchSource:0}: Error finding container d4754b6cec0099461228feaff2e73b5e052959631194b6880a7019cb08d57c62: Status 404 returned error can't find the container with id d4754b6cec0099461228feaff2e73b5e052959631194b6880a7019cb08d57c62 Oct 30 12:30:01.278934 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:01.278906 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg"] Oct 30 12:30:01.311535 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:30:01.311501 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957c1c16_b7c0_4743_927f_0152ac003fcd.slice/crio-ea93847b700e1e93484ca7cbf99ba94f2dd2ed20f55a5dbef0f8e17d02f18c9d WatchSource:0}: Error finding container ea93847b700e1e93484ca7cbf99ba94f2dd2ed20f55a5dbef0f8e17d02f18c9d: Status 404 returned error can't find the container with id ea93847b700e1e93484ca7cbf99ba94f2dd2ed20f55a5dbef0f8e17d02f18c9d Oct 30 12:30:01.313781 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:01.313750 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v"] Oct 30 12:30:02.026255 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:02.026192 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" event={"ID":"3ddcfacd-6aa3-41d2-957f-21919ee15afd","Type":"ContainerStarted","Data":"fd7361f0008df8dc15c60197260569028c80a5829d5889ccb31298d3a540550f"} Oct 30 12:30:02.026436 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:02.026306 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" event={"ID":"3ddcfacd-6aa3-41d2-957f-21919ee15afd","Type":"ContainerStarted","Data":"d4754b6cec0099461228feaff2e73b5e052959631194b6880a7019cb08d57c62"} Oct 30 12:30:02.027823 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:02.027790 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" event={"ID":"957c1c16-b7c0-4743-927f-0152ac003fcd","Type":"ContainerStarted","Data":"08d065f2291c2c928172a57794fef8c90b32fccad95340f43b8d31f6fa97b883"} Oct 30 12:30:02.027965 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:02.027844 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" event={"ID":"957c1c16-b7c0-4743-927f-0152ac003fcd","Type":"ContainerStarted","Data":"ea93847b700e1e93484ca7cbf99ba94f2dd2ed20f55a5dbef0f8e17d02f18c9d"} Oct 30 12:30:02.029396 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:02.029375 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" event={"ID":"d790e0f0-b3bc-4a27-aaaf-605ddd0e4595","Type":"ContainerStarted","Data":"02f8122a2b5151347e3dd1c9c1b27f6bf7b8787491b1938f2ac61508f00c19c1"} Oct 30 12:30:02.029476 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:02.029409 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" event={"ID":"d790e0f0-b3bc-4a27-aaaf-605ddd0e4595","Type":"ContainerStarted","Data":"a04d637d315f5c563258e545950e010b978e462698f456324766550c9ddf03cf"} Oct 30 12:30:02.070245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:02.070199 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" podStartSLOduration=2.070186065 podStartE2EDuration="2.070186065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:02.068265974 +0000 UTC m=+2153.102404382" watchObservedRunningTime="2026-10-30 12:30:02.070186065 +0000 UTC m=+2153.104324460" Oct 30 12:30:02.123304 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:02.123242 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" podStartSLOduration=2.123222028 podStartE2EDuration="2.123222028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:02.121434676 +0000 UTC m=+2153.155573074" watchObservedRunningTime="2026-10-30 12:30:02.123222028 +0000 UTC m=+2153.157360425" Oct 30 12:30:02.164332 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:02.164282 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" podStartSLOduration=2.164263986 podStartE2EDuration="2.164263986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:02.163178853 +0000 UTC m=+2153.197317250" watchObservedRunningTime="2026-10-30 12:30:02.164263986 +0000 UTC m=+2153.198402381" Oct 30 12:30:03.037620 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:03.037581 4468 generic.go:356] "Generic (PLEG): container finished" podID="d790e0f0-b3bc-4a27-aaaf-605ddd0e4595" containerID="02f8122a2b5151347e3dd1c9c1b27f6bf7b8787491b1938f2ac61508f00c19c1" exitCode=60 Oct 30 12:30:03.037800 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:03.037663 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" event={"ID":"d790e0f0-b3bc-4a27-aaaf-605ddd0e4595","Type":"ContainerDied","Data":"02f8122a2b5151347e3dd1c9c1b27f6bf7b8787491b1938f2ac61508f00c19c1"} Oct 30 12:30:03.037986 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:03.037969 4468 scope.go:122] "RemoveContainer" containerID="02f8122a2b5151347e3dd1c9c1b27f6bf7b8787491b1938f2ac61508f00c19c1" Oct 30 12:30:03.039612 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:03.039589 4468 generic.go:356] "Generic (PLEG): container finished" podID="3ddcfacd-6aa3-41d2-957f-21919ee15afd" containerID="fd7361f0008df8dc15c60197260569028c80a5829d5889ccb31298d3a540550f" exitCode=60 Oct 30 12:30:03.039738 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:03.039664 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" event={"ID":"3ddcfacd-6aa3-41d2-957f-21919ee15afd","Type":"ContainerDied","Data":"fd7361f0008df8dc15c60197260569028c80a5829d5889ccb31298d3a540550f"} Oct 30 12:30:03.039936 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:03.039916 4468 scope.go:122] "RemoveContainer" containerID="fd7361f0008df8dc15c60197260569028c80a5829d5889ccb31298d3a540550f" Oct 30 12:30:03.041393 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:03.041377 4468 generic.go:356] "Generic (PLEG): container finished" podID="957c1c16-b7c0-4743-927f-0152ac003fcd" containerID="08d065f2291c2c928172a57794fef8c90b32fccad95340f43b8d31f6fa97b883" exitCode=60 Oct 30 12:30:03.041494 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:03.041447 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" event={"ID":"957c1c16-b7c0-4743-927f-0152ac003fcd","Type":"ContainerDied","Data":"08d065f2291c2c928172a57794fef8c90b32fccad95340f43b8d31f6fa97b883"} Oct 30 12:30:03.041755 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:03.041739 4468 scope.go:122] "RemoveContainer" containerID="08d065f2291c2c928172a57794fef8c90b32fccad95340f43b8d31f6fa97b883" Oct 30 12:30:04.049750 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:04.049722 4468 generic.go:356] "Generic (PLEG): container finished" podID="3ddcfacd-6aa3-41d2-957f-21919ee15afd" containerID="30d639849591eca05447cc2094e3168bdf6bebe04cf7e60ce0815e3bcf05edb8" exitCode=60 Oct 30 12:30:04.050158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:04.049814 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" event={"ID":"3ddcfacd-6aa3-41d2-957f-21919ee15afd","Type":"ContainerDied","Data":"30d639849591eca05447cc2094e3168bdf6bebe04cf7e60ce0815e3bcf05edb8"} Oct 30 12:30:04.050158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:04.049876 4468 scope.go:122] "RemoveContainer" containerID="fd7361f0008df8dc15c60197260569028c80a5829d5889ccb31298d3a540550f" Oct 30 12:30:04.050158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:04.050143 4468 scope.go:122] "RemoveContainer" containerID="30d639849591eca05447cc2094e3168bdf6bebe04cf7e60ce0815e3bcf05edb8" Oct 30 12:30:04.050534 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:04.050508 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg_odh-ai-gateway-infra(3ddcfacd-6aa3-41d2-957f-21919ee15afd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" podUID="3ddcfacd-6aa3-41d2-957f-21919ee15afd" Oct 30 12:30:04.051797 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:04.051774 4468 generic.go:356] "Generic (PLEG): container finished" podID="957c1c16-b7c0-4743-927f-0152ac003fcd" containerID="716b70b00be169f5abe695813624790c1ad594a0a1cd611b2ffa88e9a149ddb7" exitCode=60 Oct 30 12:30:04.051883 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:04.051863 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" event={"ID":"957c1c16-b7c0-4743-927f-0152ac003fcd","Type":"ContainerDied","Data":"716b70b00be169f5abe695813624790c1ad594a0a1cd611b2ffa88e9a149ddb7"} Oct 30 12:30:04.052168 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:04.052154 4468 scope.go:122] "RemoveContainer" containerID="716b70b00be169f5abe695813624790c1ad594a0a1cd611b2ffa88e9a149ddb7" Oct 30 12:30:04.052466 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:04.052444 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v_odh-ai-gateway-infra(957c1c16-b7c0-4743-927f-0152ac003fcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" podUID="957c1c16-b7c0-4743-927f-0152ac003fcd" Oct 30 12:30:04.053835 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:04.053811 4468 generic.go:356] "Generic (PLEG): container finished" podID="d790e0f0-b3bc-4a27-aaaf-605ddd0e4595" containerID="70b53207a93bb8f7b5d18c09dd2f471b5062a6fe3b7ac48cd20f717be0b97712" exitCode=60 Oct 30 12:30:04.053935 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:04.053855 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" event={"ID":"d790e0f0-b3bc-4a27-aaaf-605ddd0e4595","Type":"ContainerDied","Data":"70b53207a93bb8f7b5d18c09dd2f471b5062a6fe3b7ac48cd20f717be0b97712"} Oct 30 12:30:04.054192 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:04.054176 4468 scope.go:122] "RemoveContainer" containerID="70b53207a93bb8f7b5d18c09dd2f471b5062a6fe3b7ac48cd20f717be0b97712" Oct 30 12:30:04.054451 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:04.054429 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2_odh-ai-gateway-infra(d790e0f0-b3bc-4a27-aaaf-605ddd0e4595)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" podUID="d790e0f0-b3bc-4a27-aaaf-605ddd0e4595" Oct 30 12:30:04.148763 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:04.148736 4468 scope.go:122] "RemoveContainer" containerID="08d065f2291c2c928172a57794fef8c90b32fccad95340f43b8d31f6fa97b883" Oct 30 12:30:04.164559 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:04.164538 4468 scope.go:122] "RemoveContainer" containerID="02f8122a2b5151347e3dd1c9c1b27f6bf7b8787491b1938f2ac61508f00c19c1" Oct 30 12:30:05.062654 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:05.062627 4468 scope.go:122] "RemoveContainer" containerID="30d639849591eca05447cc2094e3168bdf6bebe04cf7e60ce0815e3bcf05edb8" Oct 30 12:30:05.063151 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:05.062950 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg_odh-ai-gateway-infra(3ddcfacd-6aa3-41d2-957f-21919ee15afd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" podUID="3ddcfacd-6aa3-41d2-957f-21919ee15afd" Oct 30 12:30:05.064388 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:05.064358 4468 scope.go:122] "RemoveContainer" containerID="716b70b00be169f5abe695813624790c1ad594a0a1cd611b2ffa88e9a149ddb7" Oct 30 12:30:05.064633 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:05.064599 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v_odh-ai-gateway-infra(957c1c16-b7c0-4743-927f-0152ac003fcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" podUID="957c1c16-b7c0-4743-927f-0152ac003fcd" Oct 30 12:30:05.065941 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:05.065925 4468 scope.go:122] "RemoveContainer" containerID="70b53207a93bb8f7b5d18c09dd2f471b5062a6fe3b7ac48cd20f717be0b97712" Oct 30 12:30:05.066196 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:05.066179 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2_odh-ai-gateway-infra(d790e0f0-b3bc-4a27-aaaf-605ddd0e4595)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" podUID="d790e0f0-b3bc-4a27-aaaf-605ddd0e4595" Oct 30 12:30:06.047341 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:06.047307 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl"] Oct 30 12:30:06.047527 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:06.047508 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" podUID="56010c34-d070-4709-8ec7-6c9a23178226" containerName="istio-proxy" containerID="cri-o://ca8a447a91f4011e46daba356df258ffc63f96360c98a49489c6d0c9278e1683" gracePeriod=30 Oct 30 12:30:09.525175 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:09.525137 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:30:09.525944 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:09.525201 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" podUID="56010c34-d070-4709-8ec7-6c9a23178226" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:30:10.754783 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:10.754746 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6"] Oct 30 12:30:10.761835 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:10.761813 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6" Oct 30 12:30:10.776144 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:10.776111 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6"] Oct 30 12:30:10.782506 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:10.782481 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:30:10.869041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:10.869009 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615f8b23-a00e-4273-9b34-0c704075965f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-748f7476-dnbw6\" (UID: \"615f8b23-a00e-4273-9b34-0c704075965f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6" Oct 30 12:30:10.869258 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:10.869054 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msr2z\" (UniqueName: \"kubernetes.io/projected/615f8b23-a00e-4273-9b34-0c704075965f-kube-api-access-msr2z\") pod \"maas-api-revoke-sub-748f7476-dnbw6\" (UID: \"615f8b23-a00e-4273-9b34-0c704075965f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6" Oct 30 12:30:10.869258 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:10.869149 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615f8b23-a00e-4273-9b34-0c704075965f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-748f7476-dnbw6\" (UID: \"615f8b23-a00e-4273-9b34-0c704075965f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6" Oct 30 12:30:10.970905 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:10.970862 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615f8b23-a00e-4273-9b34-0c704075965f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-748f7476-dnbw6\" (UID: \"615f8b23-a00e-4273-9b34-0c704075965f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6" Oct 30 12:30:10.971122 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:10.970916 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msr2z\" (UniqueName: \"kubernetes.io/projected/615f8b23-a00e-4273-9b34-0c704075965f-kube-api-access-msr2z\") pod \"maas-api-revoke-sub-748f7476-dnbw6\" (UID: \"615f8b23-a00e-4273-9b34-0c704075965f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6" Oct 30 12:30:10.971122 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:10.970959 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615f8b23-a00e-4273-9b34-0c704075965f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-748f7476-dnbw6\" (UID: \"615f8b23-a00e-4273-9b34-0c704075965f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6" Oct 30 12:30:10.971625 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:10.971598 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615f8b23-a00e-4273-9b34-0c704075965f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-748f7476-dnbw6\" (UID: \"615f8b23-a00e-4273-9b34-0c704075965f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6" Oct 30 12:30:10.973573 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:10.973548 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615f8b23-a00e-4273-9b34-0c704075965f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-748f7476-dnbw6\" (UID: \"615f8b23-a00e-4273-9b34-0c704075965f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6" Oct 30 12:30:10.998216 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:10.998187 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msr2z\" (UniqueName: \"kubernetes.io/projected/615f8b23-a00e-4273-9b34-0c704075965f-kube-api-access-msr2z\") pod \"maas-api-revoke-sub-748f7476-dnbw6\" (UID: \"615f8b23-a00e-4273-9b34-0c704075965f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6" Oct 30 12:30:11.080420 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.080396 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6" Oct 30 12:30:11.388274 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:30:11.388246 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615f8b23_a00e_4273_9b34_0c704075965f.slice/crio-b9f07e7658b2214d0f3553185be569b0bc5d58233d543ed9b8d644120fa58ced WatchSource:0}: Error finding container b9f07e7658b2214d0f3553185be569b0bc5d58233d543ed9b8d644120fa58ced: Status 404 returned error can't find the container with id b9f07e7658b2214d0f3553185be569b0bc5d58233d543ed9b8d644120fa58ced Oct 30 12:30:11.391704 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.391646 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6"] Oct 30 12:30:11.410335 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.410314 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:30:11.510642 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.510246 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-workload-certs\") pod \"56010c34-d070-4709-8ec7-6c9a23178226\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " Oct 30 12:30:11.510642 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.510322 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56010c34-d070-4709-8ec7-6c9a23178226-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56010c34-d070-4709-8ec7-6c9a23178226-istio-podinfo\") pod \"56010c34-d070-4709-8ec7-6c9a23178226\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " Oct 30 12:30:11.510642 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.510362 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56010c34-d070-4709-8ec7-6c9a23178226-kube-api-access-jx528\" (UniqueName: \"kubernetes.io/projected/56010c34-d070-4709-8ec7-6c9a23178226-kube-api-access-jx528\") pod \"56010c34-d070-4709-8ec7-6c9a23178226\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " Oct 30 12:30:11.510642 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.510462 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-workload-socket\") pod \"56010c34-d070-4709-8ec7-6c9a23178226\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " Oct 30 12:30:11.510642 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.510491 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56010c34-d070-4709-8ec7-6c9a23178226-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56010c34-d070-4709-8ec7-6c9a23178226-istiod-ca-cert\") pod \"56010c34-d070-4709-8ec7-6c9a23178226\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " Oct 30 12:30:11.510642 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.510517 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-workload-certs" pod "56010c34-d070-4709-8ec7-6c9a23178226" (UID: "56010c34-d070-4709-8ec7-6c9a23178226"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:11.510642 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.510540 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-istio-envoy\") pod \"56010c34-d070-4709-8ec7-6c9a23178226\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " Oct 30 12:30:11.511175 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.510683 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-istio-data\") pod \"56010c34-d070-4709-8ec7-6c9a23178226\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " Oct 30 12:30:11.511175 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.510718 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-credential-socket\") pod \"56010c34-d070-4709-8ec7-6c9a23178226\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " Oct 30 12:30:11.511175 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.510874 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56010c34-d070-4709-8ec7-6c9a23178226-istio-token\" (UniqueName: \"kubernetes.io/projected/56010c34-d070-4709-8ec7-6c9a23178226-istio-token\") pod \"56010c34-d070-4709-8ec7-6c9a23178226\" (UID: \"56010c34-d070-4709-8ec7-6c9a23178226\") " Oct 30 12:30:11.511346 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.511265 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56010c34-d070-4709-8ec7-6c9a23178226-istiod-ca-cert" pod "56010c34-d070-4709-8ec7-6c9a23178226" (UID: "56010c34-d070-4709-8ec7-6c9a23178226"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:11.511604 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.511577 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-workload-socket" pod "56010c34-d070-4709-8ec7-6c9a23178226" (UID: "56010c34-d070-4709-8ec7-6c9a23178226"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:11.511764 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.511738 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-istio-data" pod "56010c34-d070-4709-8ec7-6c9a23178226" (UID: "56010c34-d070-4709-8ec7-6c9a23178226"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:11.511851 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.511756 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:11.511851 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.511781 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:11.511851 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.511786 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-credential-socket" pod "56010c34-d070-4709-8ec7-6c9a23178226" (UID: "56010c34-d070-4709-8ec7-6c9a23178226"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:11.511851 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.511794 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56010c34-d070-4709-8ec7-6c9a23178226-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:11.513388 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.513359 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56010c34-d070-4709-8ec7-6c9a23178226-kube-api-access-jx528" pod "56010c34-d070-4709-8ec7-6c9a23178226" (UID: "56010c34-d070-4709-8ec7-6c9a23178226"). InnerVolumeSpecName "kube-api-access-jx528". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:11.513983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.513956 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56010c34-d070-4709-8ec7-6c9a23178226-istio-token" pod "56010c34-d070-4709-8ec7-6c9a23178226" (UID: "56010c34-d070-4709-8ec7-6c9a23178226"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:11.514160 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.513997 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-istio-envoy" pod "56010c34-d070-4709-8ec7-6c9a23178226" (UID: "56010c34-d070-4709-8ec7-6c9a23178226"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:11.514371 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.514342 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56010c34-d070-4709-8ec7-6c9a23178226-istio-podinfo" pod "56010c34-d070-4709-8ec7-6c9a23178226" (UID: "56010c34-d070-4709-8ec7-6c9a23178226"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:30:11.613016 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.612942 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56010c34-d070-4709-8ec7-6c9a23178226-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:11.613016 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.612977 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56010c34-d070-4709-8ec7-6c9a23178226-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:11.613016 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.612987 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx528\" (UniqueName: \"kubernetes.io/projected/56010c34-d070-4709-8ec7-6c9a23178226-kube-api-access-jx528\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:11.613016 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.612997 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:11.613016 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.613009 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:11.613016 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:11.613020 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56010c34-d070-4709-8ec7-6c9a23178226-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:12.126435 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:12.126399 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6" event={"ID":"615f8b23-a00e-4273-9b34-0c704075965f","Type":"ContainerStarted","Data":"9e069b97fb837265211d9a108dd146852423275bfafc118e8e799d4cddf0a4ce"} Oct 30 12:30:12.126873 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:12.126453 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6" event={"ID":"615f8b23-a00e-4273-9b34-0c704075965f","Type":"ContainerStarted","Data":"b9f07e7658b2214d0f3553185be569b0bc5d58233d543ed9b8d644120fa58ced"} Oct 30 12:30:12.128122 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:12.128060 4468 generic.go:356] "Generic (PLEG): container finished" podID="56010c34-d070-4709-8ec7-6c9a23178226" containerID="ca8a447a91f4011e46daba356df258ffc63f96360c98a49489c6d0c9278e1683" exitCode=0 Oct 30 12:30:12.128241 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:12.128145 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" Oct 30 12:30:12.128241 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:12.128096 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" event={"ID":"56010c34-d070-4709-8ec7-6c9a23178226","Type":"ContainerDied","Data":"ca8a447a91f4011e46daba356df258ffc63f96360c98a49489c6d0c9278e1683"} Oct 30 12:30:12.128241 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:12.128229 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl" event={"ID":"56010c34-d070-4709-8ec7-6c9a23178226","Type":"ContainerDied","Data":"cd7bc9cfcdf2698cce940a334414165e069a32b6d85aa3b86c50758e7f8f0420"} Oct 30 12:30:12.128362 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:12.128252 4468 scope.go:122] "RemoveContainer" containerID="ca8a447a91f4011e46daba356df258ffc63f96360c98a49489c6d0c9278e1683" Oct 30 12:30:12.138958 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:12.138936 4468 scope.go:122] "RemoveContainer" containerID="5eda8ae7d53781a7002538e18113ad4943607ce95403afeab63c01b68a12ba20" Oct 30 12:30:12.148357 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:12.148338 4468 scope.go:122] "RemoveContainer" containerID="ca8a447a91f4011e46daba356df258ffc63f96360c98a49489c6d0c9278e1683" Oct 30 12:30:12.148591 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:12.148573 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca8a447a91f4011e46daba356df258ffc63f96360c98a49489c6d0c9278e1683\": container with ID starting with ca8a447a91f4011e46daba356df258ffc63f96360c98a49489c6d0c9278e1683 not found: ID does not exist" containerID="ca8a447a91f4011e46daba356df258ffc63f96360c98a49489c6d0c9278e1683" Oct 30 12:30:12.148634 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:12.148599 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8a447a91f4011e46daba356df258ffc63f96360c98a49489c6d0c9278e1683"} err="failed to get container status \"ca8a447a91f4011e46daba356df258ffc63f96360c98a49489c6d0c9278e1683\": rpc error: code = NotFound desc = could not find container \"ca8a447a91f4011e46daba356df258ffc63f96360c98a49489c6d0c9278e1683\": container with ID starting with ca8a447a91f4011e46daba356df258ffc63f96360c98a49489c6d0c9278e1683 not found: ID does not exist" Oct 30 12:30:12.148634 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:12.148615 4468 scope.go:122] "RemoveContainer" containerID="5eda8ae7d53781a7002538e18113ad4943607ce95403afeab63c01b68a12ba20" Oct 30 12:30:12.148817 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:12.148802 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eda8ae7d53781a7002538e18113ad4943607ce95403afeab63c01b68a12ba20\": container with ID starting with 5eda8ae7d53781a7002538e18113ad4943607ce95403afeab63c01b68a12ba20 not found: ID does not exist" containerID="5eda8ae7d53781a7002538e18113ad4943607ce95403afeab63c01b68a12ba20" Oct 30 12:30:12.148858 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:12.148822 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eda8ae7d53781a7002538e18113ad4943607ce95403afeab63c01b68a12ba20"} err="failed to get container status \"5eda8ae7d53781a7002538e18113ad4943607ce95403afeab63c01b68a12ba20\": rpc error: code = NotFound desc = could not find container \"5eda8ae7d53781a7002538e18113ad4943607ce95403afeab63c01b68a12ba20\": container with ID starting with 5eda8ae7d53781a7002538e18113ad4943607ce95403afeab63c01b68a12ba20 not found: ID does not exist" Oct 30 12:30:12.173061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:12.173016 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6" podStartSLOduration=2.173003237 podStartE2EDuration="2.173003237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:12.169128886 +0000 UTC m=+2163.203267283" watchObservedRunningTime="2026-10-30 12:30:12.173003237 +0000 UTC m=+2163.207141695" Oct 30 12:30:12.212196 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:12.212169 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl"] Oct 30 12:30:12.216798 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:12.216772 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-xvtbl"] Oct 30 12:30:13.092054 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:13.092019 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56010c34-d070-4709-8ec7-6c9a23178226" path="/var/lib/kubelet/pods/56010c34-d070-4709-8ec7-6c9a23178226/volumes" Oct 30 12:30:13.139134 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:13.139101 4468 generic.go:356] "Generic (PLEG): container finished" podID="615f8b23-a00e-4273-9b34-0c704075965f" containerID="9e069b97fb837265211d9a108dd146852423275bfafc118e8e799d4cddf0a4ce" exitCode=0 Oct 30 12:30:13.139525 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:13.139180 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6" event={"ID":"615f8b23-a00e-4273-9b34-0c704075965f","Type":"ContainerDied","Data":"9e069b97fb837265211d9a108dd146852423275bfafc118e8e799d4cddf0a4ce"} Oct 30 12:30:14.311125 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.311098 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6" Oct 30 12:30:14.454358 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.454320 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/615f8b23-a00e-4273-9b34-0c704075965f-kube-api-access-msr2z\" (UniqueName: \"kubernetes.io/projected/615f8b23-a00e-4273-9b34-0c704075965f-kube-api-access-msr2z\") pod \"615f8b23-a00e-4273-9b34-0c704075965f\" (UID: \"615f8b23-a00e-4273-9b34-0c704075965f\") " Oct 30 12:30:14.454579 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.454491 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/615f8b23-a00e-4273-9b34-0c704075965f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615f8b23-a00e-4273-9b34-0c704075965f-maas-api-xks-ca\") pod \"615f8b23-a00e-4273-9b34-0c704075965f\" (UID: \"615f8b23-a00e-4273-9b34-0c704075965f\") " Oct 30 12:30:14.454644 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.454583 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/615f8b23-a00e-4273-9b34-0c704075965f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615f8b23-a00e-4273-9b34-0c704075965f-maas-api-openshift-ca\") pod \"615f8b23-a00e-4273-9b34-0c704075965f\" (UID: \"615f8b23-a00e-4273-9b34-0c704075965f\") " Oct 30 12:30:14.454957 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.454928 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615f8b23-a00e-4273-9b34-0c704075965f-maas-api-openshift-ca" pod "615f8b23-a00e-4273-9b34-0c704075965f" (UID: "615f8b23-a00e-4273-9b34-0c704075965f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:14.455204 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.455185 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615f8b23-a00e-4273-9b34-0c704075965f-maas-api-openshift-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:14.456763 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.456735 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615f8b23-a00e-4273-9b34-0c704075965f-kube-api-access-msr2z" pod "615f8b23-a00e-4273-9b34-0c704075965f" (UID: "615f8b23-a00e-4273-9b34-0c704075965f"). InnerVolumeSpecName "kube-api-access-msr2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:14.456998 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.456978 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/615f8b23-a00e-4273-9b34-0c704075965f-maas-api-xks-ca" pod "615f8b23-a00e-4273-9b34-0c704075965f" (UID: "615f8b23-a00e-4273-9b34-0c704075965f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:30:14.557018 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.556975 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615f8b23-a00e-4273-9b34-0c704075965f-maas-api-xks-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:14.557018 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.557012 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msr2z\" (UniqueName: \"kubernetes.io/projected/615f8b23-a00e-4273-9b34-0c704075965f-kube-api-access-msr2z\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:14.865677 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.865592 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx"] Oct 30 12:30:14.866762 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.866735 4468 cpu_manager.go:414] "containerMap: removing container" podUID="56010c34-d070-4709-8ec7-6c9a23178226" containerName="istio-proxy" Oct 30 12:30:14.866902 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.866764 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56010c34-d070-4709-8ec7-6c9a23178226" containerName="istio-proxy" Oct 30 12:30:14.866902 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.866784 4468 cpu_manager.go:414] "containerMap: removing container" podUID="615f8b23-a00e-4273-9b34-0c704075965f" containerName="revoke-keys" Oct 30 12:30:14.866902 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.866793 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="615f8b23-a00e-4273-9b34-0c704075965f" containerName="revoke-keys" Oct 30 12:30:14.866902 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.866816 4468 cpu_manager.go:414] "containerMap: removing container" podUID="56010c34-d070-4709-8ec7-6c9a23178226" containerName="istio-proxy" Oct 30 12:30:14.866902 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.866824 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56010c34-d070-4709-8ec7-6c9a23178226" containerName="istio-proxy" Oct 30 12:30:14.867196 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.867088 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="56010c34-d070-4709-8ec7-6c9a23178226" containerName="istio-proxy" Oct 30 12:30:14.867196 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.867109 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="615f8b23-a00e-4273-9b34-0c704075965f" containerName="revoke-keys" Oct 30 12:30:14.867196 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.867125 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="56010c34-d070-4709-8ec7-6c9a23178226" containerName="istio-proxy" Oct 30 12:30:14.871424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.871395 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:14.896750 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.896711 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx"] Oct 30 12:30:14.902330 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.901388 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-5ce7c4-openshift-default-dockercfg-qn2g7\"" Oct 30 12:30:15.015095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:14.986973 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-workload-certs\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.016467 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.016432 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmf5v\" (UniqueName: \"kubernetes.io/projected/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-kube-api-access-mmf5v\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.034314 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.034261 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-token\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.034557 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.034540 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-envoy\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.034691 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.034678 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-workload-socket\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.034841 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.034825 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-data\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.041098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.035099 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-credential-socket\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.041098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.035188 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-podinfo\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.041098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.035238 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istiod-ca-cert\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.178733 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.178696 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-data\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.179581 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.179550 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-credential-socket\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.179787 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.179767 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-data\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.179918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.179902 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-podinfo\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.180784 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.180762 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istiod-ca-cert\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.181016 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.180997 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-workload-certs\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.181208 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.181190 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmf5v\" (UniqueName: \"kubernetes.io/projected/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-kube-api-access-mmf5v\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.181456 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.181438 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-token\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.181574 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.181555 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-envoy\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.181690 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.181678 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-workload-socket\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.182278 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.182258 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-workload-socket\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.182395 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.180282 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-credential-socket\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.183106 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.183086 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istiod-ca-cert\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.183499 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.183480 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-workload-certs\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.196672 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.196629 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6" event={"ID":"615f8b23-a00e-4273-9b34-0c704075965f","Type":"ContainerDied","Data":"b9f07e7658b2214d0f3553185be569b0bc5d58233d543ed9b8d644120fa58ced"} Oct 30 12:30:15.196988 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.196961 4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9f07e7658b2214d0f3553185be569b0bc5d58233d543ed9b8d644120fa58ced" Oct 30 12:30:15.197133 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.197111 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6" Oct 30 12:30:15.199442 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.199413 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-podinfo\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.200379 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.199056 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-envoy\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.221352 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.221317 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-token\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.222003 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.221974 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmf5v\" (UniqueName: \"kubernetes.io/projected/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-kube-api-access-mmf5v\") pod \"e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.491656 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.491567 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:15.816371 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:30:15.816344 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3098f6ce_9631_484f_a7e4_ecdcdffe3fb4.slice/crio-88f000d9685168444a2f6fd8b2debbc90b75cc1b322c1a713704321aa07e821c WatchSource:0}: Error finding container 88f000d9685168444a2f6fd8b2debbc90b75cc1b322c1a713704321aa07e821c: Status 404 returned error can't find the container with id 88f000d9685168444a2f6fd8b2debbc90b75cc1b322c1a713704321aa07e821c Oct 30 12:30:15.819249 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.819215 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:30:15.819333 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.819288 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:30:15.819333 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.819320 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:30:15.823645 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:15.823619 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx"] Oct 30 12:30:16.083122 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:16.083006 4468 scope.go:122] "RemoveContainer" containerID="70b53207a93bb8f7b5d18c09dd2f471b5062a6fe3b7ac48cd20f717be0b97712" Oct 30 12:30:16.206791 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:16.206754 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" event={"ID":"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4","Type":"ContainerStarted","Data":"9945fe7249d029828d5e8c11ddd1b29b6106cd08c4e3e88318fdf5d6dfa59ca1"} Oct 30 12:30:16.206975 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:16.206820 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" event={"ID":"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4","Type":"ContainerStarted","Data":"88f000d9685168444a2f6fd8b2debbc90b75cc1b322c1a713704321aa07e821c"} Oct 30 12:30:16.255617 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:16.255562 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" podStartSLOduration=2.255545713 podStartE2EDuration="2.255545713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:16.254364758 +0000 UTC m=+2167.288503157" watchObservedRunningTime="2026-10-30 12:30:16.255545713 +0000 UTC m=+2167.289684127" Oct 30 12:30:16.503564 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:16.494382 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:16.505742 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:16.505705 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:17.082644 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:17.082619 4468 scope.go:122] "RemoveContainer" containerID="30d639849591eca05447cc2094e3168bdf6bebe04cf7e60ce0815e3bcf05edb8" Oct 30 12:30:17.216696 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:17.216661 4468 generic.go:356] "Generic (PLEG): container finished" podID="d790e0f0-b3bc-4a27-aaaf-605ddd0e4595" containerID="350bc84509b292e5237cbe9fda5afe76b98cf7f6d0c0c4a084eed09938641edb" exitCode=60 Oct 30 12:30:17.216870 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:17.216742 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" event={"ID":"d790e0f0-b3bc-4a27-aaaf-605ddd0e4595","Type":"ContainerDied","Data":"350bc84509b292e5237cbe9fda5afe76b98cf7f6d0c0c4a084eed09938641edb"} Oct 30 12:30:17.216870 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:17.216812 4468 scope.go:122] "RemoveContainer" containerID="70b53207a93bb8f7b5d18c09dd2f471b5062a6fe3b7ac48cd20f717be0b97712" Oct 30 12:30:17.217295 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:17.217144 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:17.217841 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:17.217821 4468 scope.go:122] "RemoveContainer" containerID="350bc84509b292e5237cbe9fda5afe76b98cf7f6d0c0c4a084eed09938641edb" Oct 30 12:30:17.218141 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:17.218116 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2_odh-ai-gateway-infra(d790e0f0-b3bc-4a27-aaaf-605ddd0e4595)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" podUID="d790e0f0-b3bc-4a27-aaaf-605ddd0e4595" Oct 30 12:30:17.218439 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:17.218420 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:30:18.083256 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:18.083220 4468 scope.go:122] "RemoveContainer" containerID="716b70b00be169f5abe695813624790c1ad594a0a1cd611b2ffa88e9a149ddb7" Oct 30 12:30:18.225659 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:18.225630 4468 generic.go:356] "Generic (PLEG): container finished" podID="3ddcfacd-6aa3-41d2-957f-21919ee15afd" containerID="a38f02d655231e6e4cabe3661f8c16e99d2e1877472473596cd20e8237388d22" exitCode=60 Oct 30 12:30:18.225814 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:18.225698 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" event={"ID":"3ddcfacd-6aa3-41d2-957f-21919ee15afd","Type":"ContainerDied","Data":"a38f02d655231e6e4cabe3661f8c16e99d2e1877472473596cd20e8237388d22"} Oct 30 12:30:18.225814 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:18.225756 4468 scope.go:122] "RemoveContainer" containerID="30d639849591eca05447cc2094e3168bdf6bebe04cf7e60ce0815e3bcf05edb8" Oct 30 12:30:18.226221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:18.226194 4468 scope.go:122] "RemoveContainer" containerID="a38f02d655231e6e4cabe3661f8c16e99d2e1877472473596cd20e8237388d22" Oct 30 12:30:18.226591 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:18.226570 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg_odh-ai-gateway-infra(3ddcfacd-6aa3-41d2-957f-21919ee15afd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" podUID="3ddcfacd-6aa3-41d2-957f-21919ee15afd" Oct 30 12:30:18.273043 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:18.272964 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2"] Oct 30 12:30:18.551543 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:18.551472 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" Oct 30 12:30:18.654637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:18.654596 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d790e0f0-b3bc-4a27-aaaf-605ddd0e4595-kube-api-access-px9q8\" (UniqueName: \"kubernetes.io/projected/d790e0f0-b3bc-4a27-aaaf-605ddd0e4595-kube-api-access-px9q8\") pod \"d790e0f0-b3bc-4a27-aaaf-605ddd0e4595\" (UID: \"d790e0f0-b3bc-4a27-aaaf-605ddd0e4595\") " Oct 30 12:30:18.654829 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:18.654681 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d790e0f0-b3bc-4a27-aaaf-605ddd0e4595-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d790e0f0-b3bc-4a27-aaaf-605ddd0e4595-maas-api-service-ca\") pod \"d790e0f0-b3bc-4a27-aaaf-605ddd0e4595\" (UID: \"d790e0f0-b3bc-4a27-aaaf-605ddd0e4595\") " Oct 30 12:30:18.655050 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:18.655019 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d790e0f0-b3bc-4a27-aaaf-605ddd0e4595-maas-api-service-ca" pod "d790e0f0-b3bc-4a27-aaaf-605ddd0e4595" (UID: "d790e0f0-b3bc-4a27-aaaf-605ddd0e4595"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:18.655434 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:18.655414 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d790e0f0-b3bc-4a27-aaaf-605ddd0e4595-maas-api-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:18.656868 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:18.656836 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d790e0f0-b3bc-4a27-aaaf-605ddd0e4595-kube-api-access-px9q8" pod "d790e0f0-b3bc-4a27-aaaf-605ddd0e4595" (UID: "d790e0f0-b3bc-4a27-aaaf-605ddd0e4595"). InnerVolumeSpecName "kube-api-access-px9q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:18.756585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:18.756553 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px9q8\" (UniqueName: \"kubernetes.io/projected/d790e0f0-b3bc-4a27-aaaf-605ddd0e4595-kube-api-access-px9q8\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:19.237935 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:19.237902 4468 generic.go:356] "Generic (PLEG): container finished" podID="957c1c16-b7c0-4743-927f-0152ac003fcd" containerID="28e00e1b82da4dd155ca53057b5846fe62a2c097f58a161085f45e7459bea8a4" exitCode=60 Oct 30 12:30:19.238399 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:19.237982 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" event={"ID":"957c1c16-b7c0-4743-927f-0152ac003fcd","Type":"ContainerDied","Data":"28e00e1b82da4dd155ca53057b5846fe62a2c097f58a161085f45e7459bea8a4"} Oct 30 12:30:19.238399 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:19.238043 4468 scope.go:122] "RemoveContainer" containerID="716b70b00be169f5abe695813624790c1ad594a0a1cd611b2ffa88e9a149ddb7" Oct 30 12:30:19.238555 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:19.238536 4468 scope.go:122] "RemoveContainer" containerID="28e00e1b82da4dd155ca53057b5846fe62a2c097f58a161085f45e7459bea8a4" Oct 30 12:30:19.239491 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:19.238964 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v_odh-ai-gateway-infra(957c1c16-b7c0-4743-927f-0152ac003fcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" podUID="957c1c16-b7c0-4743-927f-0152ac003fcd" Oct 30 12:30:19.239797 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:19.239779 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" Oct 30 12:30:19.239923 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:19.239837 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2" event={"ID":"d790e0f0-b3bc-4a27-aaaf-605ddd0e4595","Type":"ContainerDied","Data":"a04d637d315f5c563258e545950e010b978e462698f456324766550c9ddf03cf"} Oct 30 12:30:19.250631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:19.250612 4468 scope.go:122] "RemoveContainer" containerID="350bc84509b292e5237cbe9fda5afe76b98cf7f6d0c0c4a084eed09938641edb" Oct 30 12:30:19.288626 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:19.288595 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg"] Oct 30 12:30:19.336526 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:19.335871 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2"] Oct 30 12:30:19.342273 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:19.342240 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f78d4-29889390-wrkk2"] Oct 30 12:30:19.480707 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:19.480678 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" Oct 30 12:30:19.671309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:19.671281 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ddcfacd-6aa3-41d2-957f-21919ee15afd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ddcfacd-6aa3-41d2-957f-21919ee15afd-maas-api-service-ca\") pod \"3ddcfacd-6aa3-41d2-957f-21919ee15afd\" (UID: \"3ddcfacd-6aa3-41d2-957f-21919ee15afd\") " Oct 30 12:30:19.671518 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:19.671325 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ddcfacd-6aa3-41d2-957f-21919ee15afd-kube-api-access-957nw\" (UniqueName: \"kubernetes.io/projected/3ddcfacd-6aa3-41d2-957f-21919ee15afd-kube-api-access-957nw\") pod \"3ddcfacd-6aa3-41d2-957f-21919ee15afd\" (UID: \"3ddcfacd-6aa3-41d2-957f-21919ee15afd\") " Oct 30 12:30:19.671723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:19.671695 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ddcfacd-6aa3-41d2-957f-21919ee15afd-maas-api-service-ca" pod "3ddcfacd-6aa3-41d2-957f-21919ee15afd" (UID: "3ddcfacd-6aa3-41d2-957f-21919ee15afd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:19.671828 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:19.671814 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ddcfacd-6aa3-41d2-957f-21919ee15afd-maas-api-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:19.673499 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:19.673475 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ddcfacd-6aa3-41d2-957f-21919ee15afd-kube-api-access-957nw" pod "3ddcfacd-6aa3-41d2-957f-21919ee15afd" (UID: "3ddcfacd-6aa3-41d2-957f-21919ee15afd"). InnerVolumeSpecName "kube-api-access-957nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:19.773176 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:19.773141 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-957nw\" (UniqueName: \"kubernetes.io/projected/3ddcfacd-6aa3-41d2-957f-21919ee15afd-kube-api-access-957nw\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:20.247900 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.247874 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" Oct 30 12:30:20.247900 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.247879 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg" event={"ID":"3ddcfacd-6aa3-41d2-957f-21919ee15afd","Type":"ContainerDied","Data":"d4754b6cec0099461228feaff2e73b5e052959631194b6880a7019cb08d57c62"} Oct 30 12:30:20.248422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.247944 4468 scope.go:122] "RemoveContainer" containerID="a38f02d655231e6e4cabe3661f8c16e99d2e1877472473596cd20e8237388d22" Oct 30 12:30:20.388092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.384131 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v"] Oct 30 12:30:20.420091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.419121 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg"] Oct 30 12:30:20.451170 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.445893 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889390-qnrzg"] Oct 30 12:30:20.674345 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.674313 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" Oct 30 12:30:20.719354 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.719316 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/957c1c16-b7c0-4743-927f-0152ac003fcd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957c1c16-b7c0-4743-927f-0152ac003fcd-maas-api-service-ca\") pod \"957c1c16-b7c0-4743-927f-0152ac003fcd\" (UID: \"957c1c16-b7c0-4743-927f-0152ac003fcd\") " Oct 30 12:30:20.719535 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.719390 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/957c1c16-b7c0-4743-927f-0152ac003fcd-kube-api-access-dqgk2\" (UniqueName: \"kubernetes.io/projected/957c1c16-b7c0-4743-927f-0152ac003fcd-kube-api-access-dqgk2\") pod \"957c1c16-b7c0-4743-927f-0152ac003fcd\" (UID: \"957c1c16-b7c0-4743-927f-0152ac003fcd\") " Oct 30 12:30:20.727107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.722940 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957c1c16-b7c0-4743-927f-0152ac003fcd-maas-api-service-ca" pod "957c1c16-b7c0-4743-927f-0152ac003fcd" (UID: "957c1c16-b7c0-4743-927f-0152ac003fcd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:20.736651 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.736616 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957c1c16-b7c0-4743-927f-0152ac003fcd-kube-api-access-dqgk2" pod "957c1c16-b7c0-4743-927f-0152ac003fcd" (UID: "957c1c16-b7c0-4743-927f-0152ac003fcd"). InnerVolumeSpecName "kube-api-access-dqgk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:20.821828 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.821714 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957c1c16-b7c0-4743-927f-0152ac003fcd-maas-api-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:20.821828 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.821749 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqgk2\" (UniqueName: \"kubernetes.io/projected/957c1c16-b7c0-4743-927f-0152ac003fcd-kube-api-access-dqgk2\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:20.952302 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.952269 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj"] Oct 30 12:30:20.952970 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.952956 4468 cpu_manager.go:414] "containerMap: removing container" podUID="d790e0f0-b3bc-4a27-aaaf-605ddd0e4595" containerName="cleanup" Oct 30 12:30:20.953015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.952972 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d790e0f0-b3bc-4a27-aaaf-605ddd0e4595" containerName="cleanup" Oct 30 12:30:20.953015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.952992 4468 cpu_manager.go:414] "containerMap: removing container" podUID="957c1c16-b7c0-4743-927f-0152ac003fcd" containerName="cleanup" Oct 30 12:30:20.953015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.952997 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957c1c16-b7c0-4743-927f-0152ac003fcd" containerName="cleanup" Oct 30 12:30:20.953015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.953010 4468 cpu_manager.go:414] "containerMap: removing container" podUID="957c1c16-b7c0-4743-927f-0152ac003fcd" containerName="cleanup" Oct 30 12:30:20.953015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.953015 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957c1c16-b7c0-4743-927f-0152ac003fcd" containerName="cleanup" Oct 30 12:30:20.953187 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.953026 4468 cpu_manager.go:414] "containerMap: removing container" podUID="d790e0f0-b3bc-4a27-aaaf-605ddd0e4595" containerName="cleanup" Oct 30 12:30:20.953187 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.953031 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d790e0f0-b3bc-4a27-aaaf-605ddd0e4595" containerName="cleanup" Oct 30 12:30:20.953187 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.953049 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3ddcfacd-6aa3-41d2-957f-21919ee15afd" containerName="cleanup" Oct 30 12:30:20.953187 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.953054 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ddcfacd-6aa3-41d2-957f-21919ee15afd" containerName="cleanup" Oct 30 12:30:20.953187 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.953060 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3ddcfacd-6aa3-41d2-957f-21919ee15afd" containerName="cleanup" Oct 30 12:30:20.953187 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.953065 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ddcfacd-6aa3-41d2-957f-21919ee15afd" containerName="cleanup" Oct 30 12:30:20.953187 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.953169 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="957c1c16-b7c0-4743-927f-0152ac003fcd" containerName="cleanup" Oct 30 12:30:20.953187 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.953179 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ddcfacd-6aa3-41d2-957f-21919ee15afd" containerName="cleanup" Oct 30 12:30:20.953187 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.953184 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ddcfacd-6aa3-41d2-957f-21919ee15afd" containerName="cleanup" Oct 30 12:30:20.953437 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.953192 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ddcfacd-6aa3-41d2-957f-21919ee15afd" containerName="cleanup" Oct 30 12:30:20.953437 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.953198 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d790e0f0-b3bc-4a27-aaaf-605ddd0e4595" containerName="cleanup" Oct 30 12:30:20.953437 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.953207 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d790e0f0-b3bc-4a27-aaaf-605ddd0e4595" containerName="cleanup" Oct 30 12:30:20.953437 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.953211 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="957c1c16-b7c0-4743-927f-0152ac003fcd" containerName="cleanup" Oct 30 12:30:20.956990 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.956962 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj" Oct 30 12:30:20.973311 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.973282 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-5ce7c4-dockercfg-rkq2j\"" Oct 30 12:30:20.973461 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.973375 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-5ce7c4\"" Oct 30 12:30:20.977861 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:20.977825 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj"] Oct 30 12:30:21.033774 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:21.028994 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zv2h\" (UniqueName: \"kubernetes.io/projected/aa3ba54c-2c7d-4274-97a4-f6335649d9e7-kube-api-access-8zv2h\") pod \"payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj\" (UID: \"aa3ba54c-2c7d-4274-97a4-f6335649d9e7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj" Oct 30 12:30:21.033774 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:21.029439 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3ba54c-2c7d-4274-97a4-f6335649d9e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj\" (UID: \"aa3ba54c-2c7d-4274-97a4-f6335649d9e7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj" Oct 30 12:30:21.179597 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:21.179347 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3ba54c-2c7d-4274-97a4-f6335649d9e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj\" (UID: \"aa3ba54c-2c7d-4274-97a4-f6335649d9e7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj" Oct 30 12:30:21.179597 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:21.179427 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zv2h\" (UniqueName: \"kubernetes.io/projected/aa3ba54c-2c7d-4274-97a4-f6335649d9e7-kube-api-access-8zv2h\") pod \"payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj\" (UID: \"aa3ba54c-2c7d-4274-97a4-f6335649d9e7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj" Oct 30 12:30:21.182921 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:21.182799 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3ba54c-2c7d-4274-97a4-f6335649d9e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj\" (UID: \"aa3ba54c-2c7d-4274-97a4-f6335649d9e7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj" Oct 30 12:30:21.244092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:21.243483 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ddcfacd-6aa3-41d2-957f-21919ee15afd" path="/var/lib/kubelet/pods/3ddcfacd-6aa3-41d2-957f-21919ee15afd/volumes" Oct 30 12:30:21.250907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:21.245531 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d790e0f0-b3bc-4a27-aaaf-605ddd0e4595" path="/var/lib/kubelet/pods/d790e0f0-b3bc-4a27-aaaf-605ddd0e4595/volumes" Oct 30 12:30:21.251731 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:21.251711 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zv2h\" (UniqueName: \"kubernetes.io/projected/aa3ba54c-2c7d-4274-97a4-f6335649d9e7-kube-api-access-8zv2h\") pod \"payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj\" (UID: \"aa3ba54c-2c7d-4274-97a4-f6335649d9e7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj" Oct 30 12:30:21.271037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:21.270471 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" Oct 30 12:30:21.271037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:21.270565 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v" event={"ID":"957c1c16-b7c0-4743-927f-0152ac003fcd","Type":"ContainerDied","Data":"ea93847b700e1e93484ca7cbf99ba94f2dd2ed20f55a5dbef0f8e17d02f18c9d"} Oct 30 12:30:21.271037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:21.270612 4468 scope.go:122] "RemoveContainer" containerID="28e00e1b82da4dd155ca53057b5846fe62a2c097f58a161085f45e7459bea8a4" Oct 30 12:30:21.271986 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:21.271968 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj" Oct 30 12:30:21.331585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:21.331543 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v"] Oct 30 12:30:21.340555 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:21.340514 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-41bdda-29889390-lgb8v"] Oct 30 12:30:21.601444 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:30:21.601415 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa3ba54c_2c7d_4274_97a4_f6335649d9e7.slice/crio-21ca8a0eb5f42bdbf5ff91d78312d4235c1f5c79fc4ab4bc8f30353ecfbdb414 WatchSource:0}: Error finding container 21ca8a0eb5f42bdbf5ff91d78312d4235c1f5c79fc4ab4bc8f30353ecfbdb414: Status 404 returned error can't find the container with id 21ca8a0eb5f42bdbf5ff91d78312d4235c1f5c79fc4ab4bc8f30353ecfbdb414 Oct 30 12:30:21.605236 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:21.605211 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj"] Oct 30 12:30:22.279792 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:22.279747 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj" event={"ID":"aa3ba54c-2c7d-4274-97a4-f6335649d9e7","Type":"ContainerStarted","Data":"56a51600ff853a1afb777a8417a4456ce7427273da2d62cf6ff23437006bb64e"} Oct 30 12:30:22.280263 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:22.279825 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj" Oct 30 12:30:22.280263 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:22.279852 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj" event={"ID":"aa3ba54c-2c7d-4274-97a4-f6335649d9e7","Type":"ContainerStarted","Data":"21ca8a0eb5f42bdbf5ff91d78312d4235c1f5c79fc4ab4bc8f30353ecfbdb414"} Oct 30 12:30:22.323152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:22.323098 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj" podStartSLOduration=2.323085007 podStartE2EDuration="2.323085007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:22.322293615 +0000 UTC m=+2173.356432011" watchObservedRunningTime="2026-10-30 12:30:22.323085007 +0000 UTC m=+2173.357223396" Oct 30 12:30:23.092465 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:23.092419 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="957c1c16-b7c0-4743-927f-0152ac003fcd" path="/var/lib/kubelet/pods/957c1c16-b7c0-4743-927f-0152ac003fcd/volumes" Oct 30 12:30:23.601873 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:23.601837 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:30:27.648305 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:27.648270 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn"] Oct 30 12:30:27.682890 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:27.682856 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8"] Oct 30 12:30:28.773825 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:28.773318 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh"] Oct 30 12:30:28.773825 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:28.773568 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" podUID="fb116259-4d49-491d-b7e8-09b1c46d9caf" containerName="maas-api" containerID="cri-o://0f110d82ea58326bd85e6c45744c688a82142410b856867f2560c048e7fc0abb" gracePeriod=30 Oct 30 12:30:29.198573 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.198222 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" Oct 30 12:30:29.239095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.238000 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb116259-4d49-491d-b7e8-09b1c46d9caf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb116259-4d49-491d-b7e8-09b1c46d9caf-maas-api-tls\") pod \"fb116259-4d49-491d-b7e8-09b1c46d9caf\" (UID: \"fb116259-4d49-491d-b7e8-09b1c46d9caf\") " Oct 30 12:30:29.239095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.238097 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb116259-4d49-491d-b7e8-09b1c46d9caf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb116259-4d49-491d-b7e8-09b1c46d9caf-metrics-certs\") pod \"fb116259-4d49-491d-b7e8-09b1c46d9caf\" (UID: \"fb116259-4d49-491d-b7e8-09b1c46d9caf\") " Oct 30 12:30:29.239095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.238260 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb116259-4d49-491d-b7e8-09b1c46d9caf-kube-api-access-rnfvp\" (UniqueName: \"kubernetes.io/projected/fb116259-4d49-491d-b7e8-09b1c46d9caf-kube-api-access-rnfvp\") pod \"fb116259-4d49-491d-b7e8-09b1c46d9caf\" (UID: \"fb116259-4d49-491d-b7e8-09b1c46d9caf\") " Oct 30 12:30:29.239095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.238327 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb116259-4d49-491d-b7e8-09b1c46d9caf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb116259-4d49-491d-b7e8-09b1c46d9caf-openshift-service-ca\") pod \"fb116259-4d49-491d-b7e8-09b1c46d9caf\" (UID: \"fb116259-4d49-491d-b7e8-09b1c46d9caf\") " Oct 30 12:30:29.274431 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.274359 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb116259-4d49-491d-b7e8-09b1c46d9caf-openshift-service-ca" pod "fb116259-4d49-491d-b7e8-09b1c46d9caf" (UID: "fb116259-4d49-491d-b7e8-09b1c46d9caf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:29.309761 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.309697 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb116259-4d49-491d-b7e8-09b1c46d9caf-metrics-certs" pod "fb116259-4d49-491d-b7e8-09b1c46d9caf" (UID: "fb116259-4d49-491d-b7e8-09b1c46d9caf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:30:29.329309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.328264 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb116259-4d49-491d-b7e8-09b1c46d9caf-maas-api-tls" pod "fb116259-4d49-491d-b7e8-09b1c46d9caf" (UID: "fb116259-4d49-491d-b7e8-09b1c46d9caf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:30:29.329309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.328914 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb116259-4d49-491d-b7e8-09b1c46d9caf-kube-api-access-rnfvp" pod "fb116259-4d49-491d-b7e8-09b1c46d9caf" (UID: "fb116259-4d49-491d-b7e8-09b1c46d9caf"). InnerVolumeSpecName "kube-api-access-rnfvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:29.341226 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.341195 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb116259-4d49-491d-b7e8-09b1c46d9caf-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:29.341439 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.341423 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnfvp\" (UniqueName: \"kubernetes.io/projected/fb116259-4d49-491d-b7e8-09b1c46d9caf-kube-api-access-rnfvp\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:29.341539 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.341527 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb116259-4d49-491d-b7e8-09b1c46d9caf-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:29.341632 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.341620 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb116259-4d49-491d-b7e8-09b1c46d9caf-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:29.377399 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.376245 4468 generic.go:356] "Generic (PLEG): container finished" podID="fb116259-4d49-491d-b7e8-09b1c46d9caf" containerID="0f110d82ea58326bd85e6c45744c688a82142410b856867f2560c048e7fc0abb" exitCode=0 Oct 30 12:30:29.377399 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.376299 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" event={"ID":"fb116259-4d49-491d-b7e8-09b1c46d9caf","Type":"ContainerDied","Data":"0f110d82ea58326bd85e6c45744c688a82142410b856867f2560c048e7fc0abb"} Oct 30 12:30:29.377399 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.376348 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" event={"ID":"fb116259-4d49-491d-b7e8-09b1c46d9caf","Type":"ContainerDied","Data":"7df6144bdc9f9c619f7597fa2d0ee051c8efe1138f62b43588fccf6b413f30af"} Oct 30 12:30:29.377399 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.376379 4468 scope.go:122] "RemoveContainer" containerID="0f110d82ea58326bd85e6c45744c688a82142410b856867f2560c048e7fc0abb" Oct 30 12:30:29.377399 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.376534 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh" Oct 30 12:30:29.401344 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.401317 4468 scope.go:122] "RemoveContainer" containerID="0f110d82ea58326bd85e6c45744c688a82142410b856867f2560c048e7fc0abb" Oct 30 12:30:29.401730 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:29.401656 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f110d82ea58326bd85e6c45744c688a82142410b856867f2560c048e7fc0abb\": container with ID starting with 0f110d82ea58326bd85e6c45744c688a82142410b856867f2560c048e7fc0abb not found: ID does not exist" containerID="0f110d82ea58326bd85e6c45744c688a82142410b856867f2560c048e7fc0abb" Oct 30 12:30:29.401730 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.401701 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f110d82ea58326bd85e6c45744c688a82142410b856867f2560c048e7fc0abb"} err="failed to get container status \"0f110d82ea58326bd85e6c45744c688a82142410b856867f2560c048e7fc0abb\": rpc error: code = NotFound desc = could not find container \"0f110d82ea58326bd85e6c45744c688a82142410b856867f2560c048e7fc0abb\": container with ID starting with 0f110d82ea58326bd85e6c45744c688a82142410b856867f2560c048e7fc0abb not found: ID does not exist" Oct 30 12:30:29.458383 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.458343 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh"] Oct 30 12:30:29.462788 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:29.462754 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ab9381-85486fcb4f-r8fmh"] Oct 30 12:30:31.093143 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:31.093105 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb116259-4d49-491d-b7e8-09b1c46d9caf" path="/var/lib/kubelet/pods/fb116259-4d49-491d-b7e8-09b1c46d9caf/volumes" Oct 30 12:30:32.648805 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:32.648762 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn" podUID="16b5b939-9d39-4175-9735-25bd51175c3d" containerName="payload-processing" containerID="cri-o://ad425d39f0b59a984f85bb145f7587f28a28c8e98ab2ce669e7b9b9cbbe52171" gracePeriod=25 Oct 30 12:30:32.683561 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:32.683526 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8" podUID="24ab8c1a-3da3-48a4-97ef-fed9c22f14ed" containerName="payload-pre-processing" containerID="cri-o://255c955e1f4ec668706140d69abe8627ff892878d8ac7d82b2303cdcebe2ed45" gracePeriod=25 Oct 30 12:30:32.981108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:32.981082 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn" Oct 30 12:30:33.023540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.023513 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8" Oct 30 12:30:33.107456 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.107427 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ab8c1a-3da3-48a4-97ef-fed9c22f14ed-kube-api-access-n5vbc\" (UniqueName: \"kubernetes.io/projected/24ab8c1a-3da3-48a4-97ef-fed9c22f14ed-kube-api-access-n5vbc\") pod \"24ab8c1a-3da3-48a4-97ef-fed9c22f14ed\" (UID: \"24ab8c1a-3da3-48a4-97ef-fed9c22f14ed\") " Oct 30 12:30:33.107627 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.107477 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b5b939-9d39-4175-9735-25bd51175c3d-kube-api-access-gbt7r\" (UniqueName: \"kubernetes.io/projected/16b5b939-9d39-4175-9735-25bd51175c3d-kube-api-access-gbt7r\") pod \"16b5b939-9d39-4175-9735-25bd51175c3d\" (UID: \"16b5b939-9d39-4175-9735-25bd51175c3d\") " Oct 30 12:30:33.107627 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.107598 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b5b939-9d39-4175-9735-25bd51175c3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b5b939-9d39-4175-9735-25bd51175c3d-plugins-config-volume\") pod \"16b5b939-9d39-4175-9735-25bd51175c3d\" (UID: \"16b5b939-9d39-4175-9735-25bd51175c3d\") " Oct 30 12:30:33.107785 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.107644 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24ab8c1a-3da3-48a4-97ef-fed9c22f14ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ab8c1a-3da3-48a4-97ef-fed9c22f14ed-plugins-config-volume\") pod \"24ab8c1a-3da3-48a4-97ef-fed9c22f14ed\" (UID: \"24ab8c1a-3da3-48a4-97ef-fed9c22f14ed\") " Oct 30 12:30:33.108322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.108285 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b5b939-9d39-4175-9735-25bd51175c3d-plugins-config-volume" pod "16b5b939-9d39-4175-9735-25bd51175c3d" (UID: "16b5b939-9d39-4175-9735-25bd51175c3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:33.108540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.108508 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ab8c1a-3da3-48a4-97ef-fed9c22f14ed-plugins-config-volume" pod "24ab8c1a-3da3-48a4-97ef-fed9c22f14ed" (UID: "24ab8c1a-3da3-48a4-97ef-fed9c22f14ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:33.110760 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.110734 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b5b939-9d39-4175-9735-25bd51175c3d-kube-api-access-gbt7r" pod "16b5b939-9d39-4175-9735-25bd51175c3d" (UID: "16b5b939-9d39-4175-9735-25bd51175c3d"). InnerVolumeSpecName "kube-api-access-gbt7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:33.110867 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.110809 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ab8c1a-3da3-48a4-97ef-fed9c22f14ed-kube-api-access-n5vbc" pod "24ab8c1a-3da3-48a4-97ef-fed9c22f14ed" (UID: "24ab8c1a-3da3-48a4-97ef-fed9c22f14ed"). InnerVolumeSpecName "kube-api-access-n5vbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:33.227276 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.226919 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5vbc\" (UniqueName: \"kubernetes.io/projected/24ab8c1a-3da3-48a4-97ef-fed9c22f14ed-kube-api-access-n5vbc\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:33.227276 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.226954 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbt7r\" (UniqueName: \"kubernetes.io/projected/16b5b939-9d39-4175-9735-25bd51175c3d-kube-api-access-gbt7r\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:33.227276 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.226969 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b5b939-9d39-4175-9735-25bd51175c3d-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:33.227276 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.226983 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ab8c1a-3da3-48a4-97ef-fed9c22f14ed-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:33.291983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.291955 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj" Oct 30 12:30:33.409237 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.409201 4468 generic.go:356] "Generic (PLEG): container finished" podID="24ab8c1a-3da3-48a4-97ef-fed9c22f14ed" containerID="255c955e1f4ec668706140d69abe8627ff892878d8ac7d82b2303cdcebe2ed45" exitCode=0 Oct 30 12:30:33.409433 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.409248 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8" Oct 30 12:30:33.409433 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.409284 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8" event={"ID":"24ab8c1a-3da3-48a4-97ef-fed9c22f14ed","Type":"ContainerDied","Data":"255c955e1f4ec668706140d69abe8627ff892878d8ac7d82b2303cdcebe2ed45"} Oct 30 12:30:33.409433 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.409342 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8" event={"ID":"24ab8c1a-3da3-48a4-97ef-fed9c22f14ed","Type":"ContainerDied","Data":"8b6e0ddc4e3b70e0505c1358eacc4dcd65bd99e1f0a0db1b43a17ce611481baa"} Oct 30 12:30:33.409433 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.409368 4468 scope.go:122] "RemoveContainer" containerID="255c955e1f4ec668706140d69abe8627ff892878d8ac7d82b2303cdcebe2ed45" Oct 30 12:30:33.411548 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.411519 4468 generic.go:356] "Generic (PLEG): container finished" podID="16b5b939-9d39-4175-9735-25bd51175c3d" containerID="ad425d39f0b59a984f85bb145f7587f28a28c8e98ab2ce669e7b9b9cbbe52171" exitCode=0 Oct 30 12:30:33.411689 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.411594 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn" Oct 30 12:30:33.411689 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.411603 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn" event={"ID":"16b5b939-9d39-4175-9735-25bd51175c3d","Type":"ContainerDied","Data":"ad425d39f0b59a984f85bb145f7587f28a28c8e98ab2ce669e7b9b9cbbe52171"} Oct 30 12:30:33.411689 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.411676 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn" event={"ID":"16b5b939-9d39-4175-9735-25bd51175c3d","Type":"ContainerDied","Data":"eb8b0f85955969034f41eaee1287e1a0a88277b8500d92d0104934861c783ab8"} Oct 30 12:30:33.421251 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.420723 4468 scope.go:122] "RemoveContainer" containerID="255c955e1f4ec668706140d69abe8627ff892878d8ac7d82b2303cdcebe2ed45" Oct 30 12:30:33.421251 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:33.421123 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255c955e1f4ec668706140d69abe8627ff892878d8ac7d82b2303cdcebe2ed45\": container with ID starting with 255c955e1f4ec668706140d69abe8627ff892878d8ac7d82b2303cdcebe2ed45 not found: ID does not exist" containerID="255c955e1f4ec668706140d69abe8627ff892878d8ac7d82b2303cdcebe2ed45" Oct 30 12:30:33.421251 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.421156 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255c955e1f4ec668706140d69abe8627ff892878d8ac7d82b2303cdcebe2ed45"} err="failed to get container status \"255c955e1f4ec668706140d69abe8627ff892878d8ac7d82b2303cdcebe2ed45\": rpc error: code = NotFound desc = could not find container \"255c955e1f4ec668706140d69abe8627ff892878d8ac7d82b2303cdcebe2ed45\": container with ID starting with 255c955e1f4ec668706140d69abe8627ff892878d8ac7d82b2303cdcebe2ed45 not found: ID does not exist" Oct 30 12:30:33.421251 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.421181 4468 scope.go:122] "RemoveContainer" containerID="ad425d39f0b59a984f85bb145f7587f28a28c8e98ab2ce669e7b9b9cbbe52171" Oct 30 12:30:33.433630 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.433608 4468 scope.go:122] "RemoveContainer" containerID="ad425d39f0b59a984f85bb145f7587f28a28c8e98ab2ce669e7b9b9cbbe52171" Oct 30 12:30:33.433924 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:33.433905 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad425d39f0b59a984f85bb145f7587f28a28c8e98ab2ce669e7b9b9cbbe52171\": container with ID starting with ad425d39f0b59a984f85bb145f7587f28a28c8e98ab2ce669e7b9b9cbbe52171 not found: ID does not exist" containerID="ad425d39f0b59a984f85bb145f7587f28a28c8e98ab2ce669e7b9b9cbbe52171" Oct 30 12:30:33.433987 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.433932 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad425d39f0b59a984f85bb145f7587f28a28c8e98ab2ce669e7b9b9cbbe52171"} err="failed to get container status \"ad425d39f0b59a984f85bb145f7587f28a28c8e98ab2ce669e7b9b9cbbe52171\": rpc error: code = NotFound desc = could not find container \"ad425d39f0b59a984f85bb145f7587f28a28c8e98ab2ce669e7b9b9cbbe52171\": container with ID starting with ad425d39f0b59a984f85bb145f7587f28a28c8e98ab2ce669e7b9b9cbbe52171 not found: ID does not exist" Oct 30 12:30:33.479204 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.473203 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn"] Oct 30 12:30:33.487113 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.487078 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ab9381-656f856d8-lhtcn"] Oct 30 12:30:33.523172 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.523133 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8"] Oct 30 12:30:33.527528 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:33.527504 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9381-548c66c6f9-hs5v8"] Oct 30 12:30:35.111151 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:35.110579 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b5b939-9d39-4175-9735-25bd51175c3d" path="/var/lib/kubelet/pods/16b5b939-9d39-4175-9735-25bd51175c3d/volumes" Oct 30 12:30:35.111661 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:35.111208 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ab8c1a-3da3-48a4-97ef-fed9c22f14ed" path="/var/lib/kubelet/pods/24ab8c1a-3da3-48a4-97ef-fed9c22f14ed/volumes" Oct 30 12:30:36.364365 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:36.364329 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst"] Oct 30 12:30:36.364802 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:36.364680 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" podUID="edfbf552-0424-4e6c-bc8d-b0334b1cb737" containerName="maas-api" containerID="cri-o://2172f1a72a30b5c938404663b7bd45bfbb285bafeb42b1d7fc80b935e1202c1d" gracePeriod=30 Oct 30 12:30:36.510190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:36.510157 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r"] Oct 30 12:30:36.510381 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:36.510359 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r" podUID="7dc9eb17-6f1d-4246-9849-9a0d75022e41" containerName="payload-processing" containerID="cri-o://ddb15156f00bbabf77f9d040a64ecd212e81223ffac43d1ab330b28f65c47394" gracePeriod=30 Oct 30 12:30:36.544561 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:36.544529 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs"] Oct 30 12:30:36.544807 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:36.544781 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs" podUID="38ad22b1-0685-49be-912b-5e72e094cb18" containerName="payload-pre-processing" containerID="cri-o://567ed6e1844117b36a5925ada2b37dd908c508e85a0cd3545cf4b18bec12b585" gracePeriod=30 Oct 30 12:30:37.012280 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.012211 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r" Oct 30 12:30:37.110228 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.110201 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" Oct 30 12:30:37.116594 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.116571 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs" Oct 30 12:30:37.129838 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.129764 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc9eb17-6f1d-4246-9849-9a0d75022e41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc9eb17-6f1d-4246-9849-9a0d75022e41-plugins-config-volume\") pod \"7dc9eb17-6f1d-4246-9849-9a0d75022e41\" (UID: \"7dc9eb17-6f1d-4246-9849-9a0d75022e41\") " Oct 30 12:30:37.129838 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.129803 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc9eb17-6f1d-4246-9849-9a0d75022e41-kube-api-access-b4q6p\" (UniqueName: \"kubernetes.io/projected/7dc9eb17-6f1d-4246-9849-9a0d75022e41-kube-api-access-b4q6p\") pod \"7dc9eb17-6f1d-4246-9849-9a0d75022e41\" (UID: \"7dc9eb17-6f1d-4246-9849-9a0d75022e41\") " Oct 30 12:30:37.130217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.130189 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc9eb17-6f1d-4246-9849-9a0d75022e41-plugins-config-volume" pod "7dc9eb17-6f1d-4246-9849-9a0d75022e41" (UID: "7dc9eb17-6f1d-4246-9849-9a0d75022e41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:37.130373 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.130353 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc9eb17-6f1d-4246-9849-9a0d75022e41-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:37.132345 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.132325 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc9eb17-6f1d-4246-9849-9a0d75022e41-kube-api-access-b4q6p" pod "7dc9eb17-6f1d-4246-9849-9a0d75022e41" (UID: "7dc9eb17-6f1d-4246-9849-9a0d75022e41"). InnerVolumeSpecName "kube-api-access-b4q6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:37.231018 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.230981 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edfbf552-0424-4e6c-bc8d-b0334b1cb737-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edfbf552-0424-4e6c-bc8d-b0334b1cb737-openshift-service-ca\") pod \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\" (UID: \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\") " Oct 30 12:30:37.231217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.231054 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edfbf552-0424-4e6c-bc8d-b0334b1cb737-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edfbf552-0424-4e6c-bc8d-b0334b1cb737-maas-api-tls\") pod \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\" (UID: \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\") " Oct 30 12:30:37.231217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.231169 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edfbf552-0424-4e6c-bc8d-b0334b1cb737-kube-api-access-qh6jr\" (UniqueName: \"kubernetes.io/projected/edfbf552-0424-4e6c-bc8d-b0334b1cb737-kube-api-access-qh6jr\") pod \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\" (UID: \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\") " Oct 30 12:30:37.231337 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.231276 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edfbf552-0424-4e6c-bc8d-b0334b1cb737-metrics-certs\" (UniqueName: \"kubernetes.io/secret/edfbf552-0424-4e6c-bc8d-b0334b1cb737-metrics-certs\") pod \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\" (UID: \"edfbf552-0424-4e6c-bc8d-b0334b1cb737\") " Oct 30 12:30:37.231401 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.231323 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edfbf552-0424-4e6c-bc8d-b0334b1cb737-openshift-service-ca" pod "edfbf552-0424-4e6c-bc8d-b0334b1cb737" (UID: "edfbf552-0424-4e6c-bc8d-b0334b1cb737"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:37.231401 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.231358 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ad22b1-0685-49be-912b-5e72e094cb18-kube-api-access-7rbgp\" (UniqueName: \"kubernetes.io/projected/38ad22b1-0685-49be-912b-5e72e094cb18-kube-api-access-7rbgp\") pod \"38ad22b1-0685-49be-912b-5e72e094cb18\" (UID: \"38ad22b1-0685-49be-912b-5e72e094cb18\") " Oct 30 12:30:37.231401 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.231390 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38ad22b1-0685-49be-912b-5e72e094cb18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ad22b1-0685-49be-912b-5e72e094cb18-plugins-config-volume\") pod \"38ad22b1-0685-49be-912b-5e72e094cb18\" (UID: \"38ad22b1-0685-49be-912b-5e72e094cb18\") " Oct 30 12:30:37.232054 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.232022 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ad22b1-0685-49be-912b-5e72e094cb18-plugins-config-volume" pod "38ad22b1-0685-49be-912b-5e72e094cb18" (UID: "38ad22b1-0685-49be-912b-5e72e094cb18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:37.232054 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.232043 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4q6p\" (UniqueName: \"kubernetes.io/projected/7dc9eb17-6f1d-4246-9849-9a0d75022e41-kube-api-access-b4q6p\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:37.232259 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.232078 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edfbf552-0424-4e6c-bc8d-b0334b1cb737-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:37.234046 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.233971 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edfbf552-0424-4e6c-bc8d-b0334b1cb737-maas-api-tls" pod "edfbf552-0424-4e6c-bc8d-b0334b1cb737" (UID: "edfbf552-0424-4e6c-bc8d-b0334b1cb737"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:30:37.234046 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.234010 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edfbf552-0424-4e6c-bc8d-b0334b1cb737-metrics-certs" pod "edfbf552-0424-4e6c-bc8d-b0334b1cb737" (UID: "edfbf552-0424-4e6c-bc8d-b0334b1cb737"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:30:37.234459 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.234440 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edfbf552-0424-4e6c-bc8d-b0334b1cb737-kube-api-access-qh6jr" pod "edfbf552-0424-4e6c-bc8d-b0334b1cb737" (UID: "edfbf552-0424-4e6c-bc8d-b0334b1cb737"). InnerVolumeSpecName "kube-api-access-qh6jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:37.235653 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.235626 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ad22b1-0685-49be-912b-5e72e094cb18-kube-api-access-7rbgp" pod "38ad22b1-0685-49be-912b-5e72e094cb18" (UID: "38ad22b1-0685-49be-912b-5e72e094cb18"). InnerVolumeSpecName "kube-api-access-7rbgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:37.333946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.333906 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edfbf552-0424-4e6c-bc8d-b0334b1cb737-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:37.333946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.333938 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh6jr\" (UniqueName: \"kubernetes.io/projected/edfbf552-0424-4e6c-bc8d-b0334b1cb737-kube-api-access-qh6jr\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:37.333946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.333947 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edfbf552-0424-4e6c-bc8d-b0334b1cb737-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:37.334228 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.333957 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rbgp\" (UniqueName: \"kubernetes.io/projected/38ad22b1-0685-49be-912b-5e72e094cb18-kube-api-access-7rbgp\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:37.334228 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.333968 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ad22b1-0685-49be-912b-5e72e094cb18-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:37.489942 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.489903 4468 generic.go:356] "Generic (PLEG): container finished" podID="38ad22b1-0685-49be-912b-5e72e094cb18" containerID="567ed6e1844117b36a5925ada2b37dd908c508e85a0cd3545cf4b18bec12b585" exitCode=0 Oct 30 12:30:37.494083 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.493892 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs" event={"ID":"38ad22b1-0685-49be-912b-5e72e094cb18","Type":"ContainerDied","Data":"567ed6e1844117b36a5925ada2b37dd908c508e85a0cd3545cf4b18bec12b585"} Oct 30 12:30:37.494244 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.493965 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs" Oct 30 12:30:37.494307 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.494156 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs" event={"ID":"38ad22b1-0685-49be-912b-5e72e094cb18","Type":"ContainerDied","Data":"c22482ae8d333d62a26e8afff62859d1230a70a510bf532ec63a0b00664ff462"} Oct 30 12:30:37.494307 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.494173 4468 scope.go:122] "RemoveContainer" containerID="567ed6e1844117b36a5925ada2b37dd908c508e85a0cd3545cf4b18bec12b585" Oct 30 12:30:37.509994 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.509975 4468 scope.go:122] "RemoveContainer" containerID="567ed6e1844117b36a5925ada2b37dd908c508e85a0cd3545cf4b18bec12b585" Oct 30 12:30:37.510470 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.510230 4468 generic.go:356] "Generic (PLEG): container finished" podID="edfbf552-0424-4e6c-bc8d-b0334b1cb737" containerID="2172f1a72a30b5c938404663b7bd45bfbb285bafeb42b1d7fc80b935e1202c1d" exitCode=0 Oct 30 12:30:37.510470 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.510282 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" Oct 30 12:30:37.510470 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:37.510372 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"567ed6e1844117b36a5925ada2b37dd908c508e85a0cd3545cf4b18bec12b585\": container with ID starting with 567ed6e1844117b36a5925ada2b37dd908c508e85a0cd3545cf4b18bec12b585 not found: ID does not exist" containerID="567ed6e1844117b36a5925ada2b37dd908c508e85a0cd3545cf4b18bec12b585" Oct 30 12:30:37.510470 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.510392 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" event={"ID":"edfbf552-0424-4e6c-bc8d-b0334b1cb737","Type":"ContainerDied","Data":"2172f1a72a30b5c938404663b7bd45bfbb285bafeb42b1d7fc80b935e1202c1d"} Oct 30 12:30:37.510470 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.510400 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567ed6e1844117b36a5925ada2b37dd908c508e85a0cd3545cf4b18bec12b585"} err="failed to get container status \"567ed6e1844117b36a5925ada2b37dd908c508e85a0cd3545cf4b18bec12b585\": rpc error: code = NotFound desc = could not find container \"567ed6e1844117b36a5925ada2b37dd908c508e85a0cd3545cf4b18bec12b585\": container with ID starting with 567ed6e1844117b36a5925ada2b37dd908c508e85a0cd3545cf4b18bec12b585 not found: ID does not exist" Oct 30 12:30:37.510470 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.510436 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst" event={"ID":"edfbf552-0424-4e6c-bc8d-b0334b1cb737","Type":"ContainerDied","Data":"36aec8d091a7a3a58982022188b0aa82cd239cab1e597aeb805388d9f0a757ee"} Oct 30 12:30:37.510470 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.510449 4468 scope.go:122] "RemoveContainer" containerID="2172f1a72a30b5c938404663b7bd45bfbb285bafeb42b1d7fc80b935e1202c1d" Oct 30 12:30:37.512411 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.512393 4468 generic.go:356] "Generic (PLEG): container finished" podID="7dc9eb17-6f1d-4246-9849-9a0d75022e41" containerID="ddb15156f00bbabf77f9d040a64ecd212e81223ffac43d1ab330b28f65c47394" exitCode=0 Oct 30 12:30:37.512521 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.512432 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r" event={"ID":"7dc9eb17-6f1d-4246-9849-9a0d75022e41","Type":"ContainerDied","Data":"ddb15156f00bbabf77f9d040a64ecd212e81223ffac43d1ab330b28f65c47394"} Oct 30 12:30:37.512521 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.512467 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r" Oct 30 12:30:37.512521 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.512486 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r" event={"ID":"7dc9eb17-6f1d-4246-9849-9a0d75022e41","Type":"ContainerDied","Data":"d76b4c9dd4afa1920ef84f103482548b9d552ba6b805dbc2b9e840f3dac1d365"} Oct 30 12:30:37.524585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.524561 4468 scope.go:122] "RemoveContainer" containerID="2172f1a72a30b5c938404663b7bd45bfbb285bafeb42b1d7fc80b935e1202c1d" Oct 30 12:30:37.524852 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:37.524831 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2172f1a72a30b5c938404663b7bd45bfbb285bafeb42b1d7fc80b935e1202c1d\": container with ID starting with 2172f1a72a30b5c938404663b7bd45bfbb285bafeb42b1d7fc80b935e1202c1d not found: ID does not exist" containerID="2172f1a72a30b5c938404663b7bd45bfbb285bafeb42b1d7fc80b935e1202c1d" Oct 30 12:30:37.524941 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.524870 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2172f1a72a30b5c938404663b7bd45bfbb285bafeb42b1d7fc80b935e1202c1d"} err="failed to get container status \"2172f1a72a30b5c938404663b7bd45bfbb285bafeb42b1d7fc80b935e1202c1d\": rpc error: code = NotFound desc = could not find container \"2172f1a72a30b5c938404663b7bd45bfbb285bafeb42b1d7fc80b935e1202c1d\": container with ID starting with 2172f1a72a30b5c938404663b7bd45bfbb285bafeb42b1d7fc80b935e1202c1d not found: ID does not exist" Oct 30 12:30:37.524941 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.524894 4468 scope.go:122] "RemoveContainer" containerID="ddb15156f00bbabf77f9d040a64ecd212e81223ffac43d1ab330b28f65c47394" Oct 30 12:30:37.551717 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.551699 4468 scope.go:122] "RemoveContainer" containerID="ddb15156f00bbabf77f9d040a64ecd212e81223ffac43d1ab330b28f65c47394" Oct 30 12:30:37.552006 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:37.551987 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddb15156f00bbabf77f9d040a64ecd212e81223ffac43d1ab330b28f65c47394\": container with ID starting with ddb15156f00bbabf77f9d040a64ecd212e81223ffac43d1ab330b28f65c47394 not found: ID does not exist" containerID="ddb15156f00bbabf77f9d040a64ecd212e81223ffac43d1ab330b28f65c47394" Oct 30 12:30:37.552111 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.552013 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb15156f00bbabf77f9d040a64ecd212e81223ffac43d1ab330b28f65c47394"} err="failed to get container status \"ddb15156f00bbabf77f9d040a64ecd212e81223ffac43d1ab330b28f65c47394\": rpc error: code = NotFound desc = could not find container \"ddb15156f00bbabf77f9d040a64ecd212e81223ffac43d1ab330b28f65c47394\": container with ID starting with ddb15156f00bbabf77f9d040a64ecd212e81223ffac43d1ab330b28f65c47394 not found: ID does not exist" Oct 30 12:30:37.595557 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.595510 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs"] Oct 30 12:30:37.600021 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.599994 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3112-697b5b645f-67mvs"] Oct 30 12:30:37.676925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.676893 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst"] Oct 30 12:30:37.681419 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.681393 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3112-767dd87b85-nkjst"] Oct 30 12:30:37.714520 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.714488 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r"] Oct 30 12:30:37.718887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:37.718862 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3112-79d664bddb-92v9r"] Oct 30 12:30:38.159643 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:38.159604 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6"] Oct 30 12:30:38.159907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:38.159875 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" podUID="88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" containerName="istio-proxy" containerID="cri-o://191b9aa92fa6fcda99c74542f4deaad0e3c5ab019d41d547b780b8b05f2c5e4a" gracePeriod=30 Oct 30 12:30:38.451036 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:38.450955 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8"] Oct 30 12:30:38.451272 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:38.451242 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" podUID="a0c16fb3-1a1a-4e66-a390-7004db9a2bec" containerName="istio-proxy" containerID="cri-o://e0abe153c69d7af6abe51e908be8fa6a5b6889117425ebc4fc4a58f24e18373d" gracePeriod=30 Oct 30 12:30:39.100879 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:39.100835 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38ad22b1-0685-49be-912b-5e72e094cb18" path="/var/lib/kubelet/pods/38ad22b1-0685-49be-912b-5e72e094cb18/volumes" Oct 30 12:30:39.101312 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:39.101297 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dc9eb17-6f1d-4246-9849-9a0d75022e41" path="/var/lib/kubelet/pods/7dc9eb17-6f1d-4246-9849-9a0d75022e41/volumes" Oct 30 12:30:39.101698 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:39.101685 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edfbf552-0424-4e6c-bc8d-b0334b1cb737" path="/var/lib/kubelet/pods/edfbf552-0424-4e6c-bc8d-b0334b1cb737/volumes" Oct 30 12:30:42.941356 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:42.941313 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj"] Oct 30 12:30:43.450738 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.450713 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:30:43.527668 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.527631 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-credential-socket\") pod \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " Oct 30 12:30:43.527836 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.527814 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-workload-socket\") pod \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " Oct 30 12:30:43.527918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.527900 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-kube-api-access-tjbts\" (UniqueName: \"kubernetes.io/projected/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-kube-api-access-tjbts\") pod \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " Oct 30 12:30:43.527981 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.527922 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-credential-socket" pod "88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" (UID: "88f3f4ba-2b76-43e2-905f-25dd92a8f2a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:43.527981 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.527974 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-podinfo\") pod \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " Oct 30 12:30:43.528143 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.528018 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-data\") pod \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " Oct 30 12:30:43.528143 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.528110 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-token\" (UniqueName: \"kubernetes.io/projected/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-token\") pod \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " Oct 30 12:30:43.528260 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.528149 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istiod-ca-cert\") pod \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " Oct 30 12:30:43.528260 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.528177 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-workload-certs\") pod \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " Oct 30 12:30:43.528260 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.528184 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-workload-socket" pod "88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" (UID: "88f3f4ba-2b76-43e2-905f-25dd92a8f2a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:43.528260 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.528230 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-envoy\") pod \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\" (UID: \"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4\") " Oct 30 12:30:43.528470 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.528383 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-data" pod "88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" (UID: "88f3f4ba-2b76-43e2-905f-25dd92a8f2a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:43.528717 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.528656 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istiod-ca-cert" pod "88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" (UID: "88f3f4ba-2b76-43e2-905f-25dd92a8f2a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:43.528928 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.528902 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:43.529020 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.528932 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:43.529020 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.528947 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:43.529020 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.528962 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:43.533111 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.530484 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-workload-certs" pod "88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" (UID: "88f3f4ba-2b76-43e2-905f-25dd92a8f2a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:43.533111 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.531462 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-podinfo" pod "88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" (UID: "88f3f4ba-2b76-43e2-905f-25dd92a8f2a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:30:43.533111 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.532249 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-token" pod "88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" (UID: "88f3f4ba-2b76-43e2-905f-25dd92a8f2a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:43.533111 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.532318 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-envoy" pod "88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" (UID: "88f3f4ba-2b76-43e2-905f-25dd92a8f2a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:43.533111 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.532391 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-kube-api-access-tjbts" pod "88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" (UID: "88f3f4ba-2b76-43e2-905f-25dd92a8f2a4"). InnerVolumeSpecName "kube-api-access-tjbts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:43.563630 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.562397 4468 generic.go:356] "Generic (PLEG): container finished" podID="88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" containerID="191b9aa92fa6fcda99c74542f4deaad0e3c5ab019d41d547b780b8b05f2c5e4a" exitCode=0 Oct 30 12:30:43.563630 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.562486 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" event={"ID":"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4","Type":"ContainerDied","Data":"191b9aa92fa6fcda99c74542f4deaad0e3c5ab019d41d547b780b8b05f2c5e4a"} Oct 30 12:30:43.563630 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.562557 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" event={"ID":"88f3f4ba-2b76-43e2-905f-25dd92a8f2a4","Type":"ContainerDied","Data":"ad01a896dfed75e31cd8e3ab38fe7ab76a406c63197a33bfa7713193dafff0e8"} Oct 30 12:30:43.563630 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.562583 4468 scope.go:122] "RemoveContainer" containerID="191b9aa92fa6fcda99c74542f4deaad0e3c5ab019d41d547b780b8b05f2c5e4a" Oct 30 12:30:43.563630 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.562711 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6" Oct 30 12:30:43.574662 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.574619 4468 scope.go:122] "RemoveContainer" containerID="660bc54ecfc8ebbf9214fd099a98bf6167e1e0349d549ca65da2d010150c4b5b" Oct 30 12:30:43.585061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.585034 4468 scope.go:122] "RemoveContainer" containerID="191b9aa92fa6fcda99c74542f4deaad0e3c5ab019d41d547b780b8b05f2c5e4a" Oct 30 12:30:43.585360 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:43.585338 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191b9aa92fa6fcda99c74542f4deaad0e3c5ab019d41d547b780b8b05f2c5e4a\": container with ID starting with 191b9aa92fa6fcda99c74542f4deaad0e3c5ab019d41d547b780b8b05f2c5e4a not found: ID does not exist" containerID="191b9aa92fa6fcda99c74542f4deaad0e3c5ab019d41d547b780b8b05f2c5e4a" Oct 30 12:30:43.585442 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.585369 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191b9aa92fa6fcda99c74542f4deaad0e3c5ab019d41d547b780b8b05f2c5e4a"} err="failed to get container status \"191b9aa92fa6fcda99c74542f4deaad0e3c5ab019d41d547b780b8b05f2c5e4a\": rpc error: code = NotFound desc = could not find container \"191b9aa92fa6fcda99c74542f4deaad0e3c5ab019d41d547b780b8b05f2c5e4a\": container with ID starting with 191b9aa92fa6fcda99c74542f4deaad0e3c5ab019d41d547b780b8b05f2c5e4a not found: ID does not exist" Oct 30 12:30:43.585442 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.585386 4468 scope.go:122] "RemoveContainer" containerID="660bc54ecfc8ebbf9214fd099a98bf6167e1e0349d549ca65da2d010150c4b5b" Oct 30 12:30:43.585635 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:43.585617 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660bc54ecfc8ebbf9214fd099a98bf6167e1e0349d549ca65da2d010150c4b5b\": container with ID starting with 660bc54ecfc8ebbf9214fd099a98bf6167e1e0349d549ca65da2d010150c4b5b not found: ID does not exist" containerID="660bc54ecfc8ebbf9214fd099a98bf6167e1e0349d549ca65da2d010150c4b5b" Oct 30 12:30:43.585698 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.585639 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660bc54ecfc8ebbf9214fd099a98bf6167e1e0349d549ca65da2d010150c4b5b"} err="failed to get container status \"660bc54ecfc8ebbf9214fd099a98bf6167e1e0349d549ca65da2d010150c4b5b\": rpc error: code = NotFound desc = could not find container \"660bc54ecfc8ebbf9214fd099a98bf6167e1e0349d549ca65da2d010150c4b5b\": container with ID starting with 660bc54ecfc8ebbf9214fd099a98bf6167e1e0349d549ca65da2d010150c4b5b not found: ID does not exist" Oct 30 12:30:43.621417 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.621383 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6"] Oct 30 12:30:43.635115 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.629616 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-ab9381-openshift-default-c9868bbc7-tjkw6"] Oct 30 12:30:43.635115 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.630498 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:43.635115 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.630531 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjbts\" (UniqueName: \"kubernetes.io/projected/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-kube-api-access-tjbts\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:43.635115 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.630562 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:43.635115 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.630572 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:43.635115 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.630580 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:43.714324 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.714298 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:30:43.832294 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.832213 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-podinfo\") pod \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " Oct 30 12:30:43.832448 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.832317 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istiod-ca-cert\") pod \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " Oct 30 12:30:43.832448 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.832434 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-envoy\") pod \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " Oct 30 12:30:43.832535 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.832476 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-data\") pod \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " Oct 30 12:30:43.832535 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.832505 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-token\" (UniqueName: \"kubernetes.io/projected/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-token\") pod \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " Oct 30 12:30:43.832639 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.832541 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-workload-certs\") pod \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " Oct 30 12:30:43.832639 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.832583 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-credential-socket\") pod \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " Oct 30 12:30:43.832759 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.832654 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-kube-api-access-shzlx\" (UniqueName: \"kubernetes.io/projected/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-kube-api-access-shzlx\") pod \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " Oct 30 12:30:43.832759 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.832723 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-workload-socket\") pod \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\" (UID: \"a0c16fb3-1a1a-4e66-a390-7004db9a2bec\") " Oct 30 12:30:43.832874 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.832749 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istiod-ca-cert" pod "a0c16fb3-1a1a-4e66-a390-7004db9a2bec" (UID: "a0c16fb3-1a1a-4e66-a390-7004db9a2bec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:43.832874 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.832773 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-data" pod "a0c16fb3-1a1a-4e66-a390-7004db9a2bec" (UID: "a0c16fb3-1a1a-4e66-a390-7004db9a2bec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:43.833104 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.833049 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-credential-socket" pod "a0c16fb3-1a1a-4e66-a390-7004db9a2bec" (UID: "a0c16fb3-1a1a-4e66-a390-7004db9a2bec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:43.833344 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.833296 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-workload-socket" pod "a0c16fb3-1a1a-4e66-a390-7004db9a2bec" (UID: "a0c16fb3-1a1a-4e66-a390-7004db9a2bec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:43.833508 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.833452 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:43.833619 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.833513 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:43.833619 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.833530 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:43.833619 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.833545 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:43.833795 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.833705 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-workload-certs" pod "a0c16fb3-1a1a-4e66-a390-7004db9a2bec" (UID: "a0c16fb3-1a1a-4e66-a390-7004db9a2bec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:43.834754 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.834730 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-envoy" pod "a0c16fb3-1a1a-4e66-a390-7004db9a2bec" (UID: "a0c16fb3-1a1a-4e66-a390-7004db9a2bec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:43.834939 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.834923 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-kube-api-access-shzlx" pod "a0c16fb3-1a1a-4e66-a390-7004db9a2bec" (UID: "a0c16fb3-1a1a-4e66-a390-7004db9a2bec"). InnerVolumeSpecName "kube-api-access-shzlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:43.835008 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.834995 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-token" pod "a0c16fb3-1a1a-4e66-a390-7004db9a2bec" (UID: "a0c16fb3-1a1a-4e66-a390-7004db9a2bec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:43.835401 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.835380 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-podinfo" pod "a0c16fb3-1a1a-4e66-a390-7004db9a2bec" (UID: "a0c16fb3-1a1a-4e66-a390-7004db9a2bec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:30:43.935345 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.935311 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:43.935345 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.935340 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:43.935345 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.935351 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:43.935578 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.935361 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shzlx\" (UniqueName: \"kubernetes.io/projected/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-kube-api-access-shzlx\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:43.935578 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:43.935372 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c16fb3-1a1a-4e66-a390-7004db9a2bec-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:44.232470 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.232433 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx"] Oct 30 12:30:44.233253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233230 4468 cpu_manager.go:414] "containerMap: removing container" podUID="edfbf552-0424-4e6c-bc8d-b0334b1cb737" containerName="maas-api" Oct 30 12:30:44.233253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233253 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edfbf552-0424-4e6c-bc8d-b0334b1cb737" containerName="maas-api" Oct 30 12:30:44.233424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233275 4468 cpu_manager.go:414] "containerMap: removing container" podUID="a0c16fb3-1a1a-4e66-a390-7004db9a2bec" containerName="istio-proxy" Oct 30 12:30:44.233424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233283 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c16fb3-1a1a-4e66-a390-7004db9a2bec" containerName="istio-proxy" Oct 30 12:30:44.233424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233296 4468 cpu_manager.go:414] "containerMap: removing container" podUID="38ad22b1-0685-49be-912b-5e72e094cb18" containerName="payload-pre-processing" Oct 30 12:30:44.233424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233303 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ad22b1-0685-49be-912b-5e72e094cb18" containerName="payload-pre-processing" Oct 30 12:30:44.233424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233318 4468 cpu_manager.go:414] "containerMap: removing container" podUID="24ab8c1a-3da3-48a4-97ef-fed9c22f14ed" containerName="payload-pre-processing" Oct 30 12:30:44.233424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233325 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ab8c1a-3da3-48a4-97ef-fed9c22f14ed" containerName="payload-pre-processing" Oct 30 12:30:44.233424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233342 4468 cpu_manager.go:414] "containerMap: removing container" podUID="957c1c16-b7c0-4743-927f-0152ac003fcd" containerName="cleanup" Oct 30 12:30:44.233424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233349 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957c1c16-b7c0-4743-927f-0152ac003fcd" containerName="cleanup" Oct 30 12:30:44.233424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233375 4468 cpu_manager.go:414] "containerMap: removing container" podUID="16b5b939-9d39-4175-9735-25bd51175c3d" containerName="payload-processing" Oct 30 12:30:44.233424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233406 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b5b939-9d39-4175-9735-25bd51175c3d" containerName="payload-processing" Oct 30 12:30:44.233424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233422 4468 cpu_manager.go:414] "containerMap: removing container" podUID="88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" containerName="istio-proxy" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233429 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" containerName="istio-proxy" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233447 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3ddcfacd-6aa3-41d2-957f-21919ee15afd" containerName="cleanup" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233455 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ddcfacd-6aa3-41d2-957f-21919ee15afd" containerName="cleanup" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233484 4468 cpu_manager.go:414] "containerMap: removing container" podUID="d790e0f0-b3bc-4a27-aaaf-605ddd0e4595" containerName="cleanup" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233491 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d790e0f0-b3bc-4a27-aaaf-605ddd0e4595" containerName="cleanup" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233505 4468 cpu_manager.go:414] "containerMap: removing container" podUID="7dc9eb17-6f1d-4246-9849-9a0d75022e41" containerName="payload-processing" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233513 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc9eb17-6f1d-4246-9849-9a0d75022e41" containerName="payload-processing" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233527 4468 cpu_manager.go:414] "containerMap: removing container" podUID="fb116259-4d49-491d-b7e8-09b1c46d9caf" containerName="maas-api" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233533 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb116259-4d49-491d-b7e8-09b1c46d9caf" containerName="maas-api" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233542 4468 cpu_manager.go:414] "containerMap: removing container" podUID="88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" containerName="istio-proxy" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233550 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" containerName="istio-proxy" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233724 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c16fb3-1a1a-4e66-a390-7004db9a2bec" containerName="istio-proxy" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233740 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ad22b1-0685-49be-912b-5e72e094cb18" containerName="payload-pre-processing" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233749 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ab8c1a-3da3-48a4-97ef-fed9c22f14ed" containerName="payload-pre-processing" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233760 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d790e0f0-b3bc-4a27-aaaf-605ddd0e4595" containerName="cleanup" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233775 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b5b939-9d39-4175-9735-25bd51175c3d" containerName="payload-processing" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233784 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="957c1c16-b7c0-4743-927f-0152ac003fcd" containerName="cleanup" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233793 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" containerName="istio-proxy" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233807 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc9eb17-6f1d-4246-9849-9a0d75022e41" containerName="payload-processing" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233816 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb116259-4d49-491d-b7e8-09b1c46d9caf" containerName="maas-api" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233824 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" containerName="istio-proxy" Oct 30 12:30:44.233949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.233836 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="edfbf552-0424-4e6c-bc8d-b0334b1cb737" containerName="maas-api" Oct 30 12:30:44.238940 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.238917 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" Oct 30 12:30:44.256105 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.255271 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-5ce7c4-serving-cert\"" Oct 30 12:30:44.260682 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.260659 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx"] Oct 30 12:30:44.340986 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.340950 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3162c67-d852-4c92-8d21-6f85009feaff-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx\" (UID: \"d3162c67-d852-4c92-8d21-6f85009feaff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" Oct 30 12:30:44.341197 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.341019 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3162c67-d852-4c92-8d21-6f85009feaff-metrics-certs\") pod \"maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx\" (UID: \"d3162c67-d852-4c92-8d21-6f85009feaff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" Oct 30 12:30:44.341197 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.341150 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st82v\" (UniqueName: \"kubernetes.io/projected/d3162c67-d852-4c92-8d21-6f85009feaff-kube-api-access-st82v\") pod \"maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx\" (UID: \"d3162c67-d852-4c92-8d21-6f85009feaff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" Oct 30 12:30:44.341314 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.341287 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3162c67-d852-4c92-8d21-6f85009feaff-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx\" (UID: \"d3162c67-d852-4c92-8d21-6f85009feaff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" Oct 30 12:30:44.455893 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.455793 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3162c67-d852-4c92-8d21-6f85009feaff-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx\" (UID: \"d3162c67-d852-4c92-8d21-6f85009feaff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" Oct 30 12:30:44.457671 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.457643 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3162c67-d852-4c92-8d21-6f85009feaff-metrics-certs\") pod \"maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx\" (UID: \"d3162c67-d852-4c92-8d21-6f85009feaff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" Oct 30 12:30:44.458412 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.458387 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st82v\" (UniqueName: \"kubernetes.io/projected/d3162c67-d852-4c92-8d21-6f85009feaff-kube-api-access-st82v\") pod \"maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx\" (UID: \"d3162c67-d852-4c92-8d21-6f85009feaff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" Oct 30 12:30:44.459923 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.459779 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3162c67-d852-4c92-8d21-6f85009feaff-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx\" (UID: \"d3162c67-d852-4c92-8d21-6f85009feaff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" Oct 30 12:30:44.460173 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.460139 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3162c67-d852-4c92-8d21-6f85009feaff-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx\" (UID: \"d3162c67-d852-4c92-8d21-6f85009feaff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" Oct 30 12:30:44.482724 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.476708 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3162c67-d852-4c92-8d21-6f85009feaff-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx\" (UID: \"d3162c67-d852-4c92-8d21-6f85009feaff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" Oct 30 12:30:44.506907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.506363 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st82v\" (UniqueName: \"kubernetes.io/projected/d3162c67-d852-4c92-8d21-6f85009feaff-kube-api-access-st82v\") pod \"maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx\" (UID: \"d3162c67-d852-4c92-8d21-6f85009feaff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" Oct 30 12:30:44.506907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.506853 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3162c67-d852-4c92-8d21-6f85009feaff-metrics-certs\") pod \"maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx\" (UID: \"d3162c67-d852-4c92-8d21-6f85009feaff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" Oct 30 12:30:44.553692 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.553659 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" Oct 30 12:30:44.571929 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.571901 4468 generic.go:356] "Generic (PLEG): container finished" podID="a0c16fb3-1a1a-4e66-a390-7004db9a2bec" containerID="e0abe153c69d7af6abe51e908be8fa6a5b6889117425ebc4fc4a58f24e18373d" exitCode=0 Oct 30 12:30:44.572041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.571961 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" Oct 30 12:30:44.572041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.571981 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" event={"ID":"a0c16fb3-1a1a-4e66-a390-7004db9a2bec","Type":"ContainerDied","Data":"e0abe153c69d7af6abe51e908be8fa6a5b6889117425ebc4fc4a58f24e18373d"} Oct 30 12:30:44.572170 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.572047 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8" event={"ID":"a0c16fb3-1a1a-4e66-a390-7004db9a2bec","Type":"ContainerDied","Data":"f361a399152e1c6beee03c584fcdd0a8db90d727747c8669d6cf0bd218150331"} Oct 30 12:30:44.572170 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.572115 4468 scope.go:122] "RemoveContainer" containerID="e0abe153c69d7af6abe51e908be8fa6a5b6889117425ebc4fc4a58f24e18373d" Oct 30 12:30:44.590666 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.590646 4468 scope.go:122] "RemoveContainer" containerID="e0abe153c69d7af6abe51e908be8fa6a5b6889117425ebc4fc4a58f24e18373d" Oct 30 12:30:44.591190 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:44.591051 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0abe153c69d7af6abe51e908be8fa6a5b6889117425ebc4fc4a58f24e18373d\": container with ID starting with e0abe153c69d7af6abe51e908be8fa6a5b6889117425ebc4fc4a58f24e18373d not found: ID does not exist" containerID="e0abe153c69d7af6abe51e908be8fa6a5b6889117425ebc4fc4a58f24e18373d" Oct 30 12:30:44.591190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.591131 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0abe153c69d7af6abe51e908be8fa6a5b6889117425ebc4fc4a58f24e18373d"} err="failed to get container status \"e0abe153c69d7af6abe51e908be8fa6a5b6889117425ebc4fc4a58f24e18373d\": rpc error: code = NotFound desc = could not find container \"e0abe153c69d7af6abe51e908be8fa6a5b6889117425ebc4fc4a58f24e18373d\": container with ID starting with e0abe153c69d7af6abe51e908be8fa6a5b6889117425ebc4fc4a58f24e18373d not found: ID does not exist" Oct 30 12:30:44.634726 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.634676 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8"] Oct 30 12:30:44.640096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.640026 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-77lx8"] Oct 30 12:30:44.869649 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:30:44.869617 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3162c67_d852_4c92_8d21_6f85009feaff.slice/crio-f23e31e88ac4268fca055b1cca73238f5c82c2ff5abca8c4b187849be6e9977b WatchSource:0}: Error finding container f23e31e88ac4268fca055b1cca73238f5c82c2ff5abca8c4b187849be6e9977b: Status 404 returned error can't find the container with id f23e31e88ac4268fca055b1cca73238f5c82c2ff5abca8c4b187849be6e9977b Oct 30 12:30:44.870505 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.870485 4468 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:30:44.873099 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:44.873041 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx"] Oct 30 12:30:45.091051 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:45.091018 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88f3f4ba-2b76-43e2-905f-25dd92a8f2a4" path="/var/lib/kubelet/pods/88f3f4ba-2b76-43e2-905f-25dd92a8f2a4/volumes" Oct 30 12:30:45.091593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:45.091577 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c16fb3-1a1a-4e66-a390-7004db9a2bec" path="/var/lib/kubelet/pods/a0c16fb3-1a1a-4e66-a390-7004db9a2bec/volumes" Oct 30 12:30:45.583508 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:45.583411 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" event={"ID":"d3162c67-d852-4c92-8d21-6f85009feaff","Type":"ContainerStarted","Data":"414288ee42949906afc17a0d00b0919f89335b9a5969c6fd2bc72af5be9cf3d1"} Oct 30 12:30:45.583508 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:45.583461 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" event={"ID":"d3162c67-d852-4c92-8d21-6f85009feaff","Type":"ContainerStarted","Data":"f23e31e88ac4268fca055b1cca73238f5c82c2ff5abca8c4b187849be6e9977b"} Oct 30 12:30:45.583936 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:45.583533 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" Oct 30 12:30:45.629807 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:45.629752 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" podStartSLOduration=1.26626715 podStartE2EDuration="1.629736668s" totalImagesPullingTime="363.469518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:45.62860373 +0000 UTC m=+2196.662742138" watchObservedRunningTime="2026-10-30 12:30:45.629736668 +0000 UTC m=+2196.663875067" Oct 30 12:30:48.128961 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:48.128925 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46"] Oct 30 12:30:48.129404 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:48.129141 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46" podUID="d11d3c04-4239-41d4-a099-3c6f0d287eaf" containerName="payload-processing" containerID="cri-o://0da1af733c106e8aa2bd0db13a67b82374ffa968846291e2afa32aa1e37cf3f4" gracePeriod=30 Oct 30 12:30:48.406736 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:48.406710 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46" Oct 30 12:30:48.541977 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:48.541938 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d11d3c04-4239-41d4-a099-3c6f0d287eaf-kube-api-access-9rhws\" (UniqueName: \"kubernetes.io/projected/d11d3c04-4239-41d4-a099-3c6f0d287eaf-kube-api-access-9rhws\") pod \"d11d3c04-4239-41d4-a099-3c6f0d287eaf\" (UID: \"d11d3c04-4239-41d4-a099-3c6f0d287eaf\") " Oct 30 12:30:48.543147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:48.543122 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d11d3c04-4239-41d4-a099-3c6f0d287eaf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11d3c04-4239-41d4-a099-3c6f0d287eaf-plugins-config-volume\") pod \"d11d3c04-4239-41d4-a099-3c6f0d287eaf\" (UID: \"d11d3c04-4239-41d4-a099-3c6f0d287eaf\") " Oct 30 12:30:48.567089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:48.543493 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d11d3c04-4239-41d4-a099-3c6f0d287eaf-plugins-config-volume" pod "d11d3c04-4239-41d4-a099-3c6f0d287eaf" (UID: "d11d3c04-4239-41d4-a099-3c6f0d287eaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:48.580350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:48.577763 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11d3c04-4239-41d4-a099-3c6f0d287eaf-kube-api-access-9rhws" pod "d11d3c04-4239-41d4-a099-3c6f0d287eaf" (UID: "d11d3c04-4239-41d4-a099-3c6f0d287eaf"). InnerVolumeSpecName "kube-api-access-9rhws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:48.618903 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:48.618865 4468 generic.go:356] "Generic (PLEG): container finished" podID="d11d3c04-4239-41d4-a099-3c6f0d287eaf" containerID="0da1af733c106e8aa2bd0db13a67b82374ffa968846291e2afa32aa1e37cf3f4" exitCode=0 Oct 30 12:30:48.619437 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:48.619280 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46" Oct 30 12:30:48.622563 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:48.619303 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46" event={"ID":"d11d3c04-4239-41d4-a099-3c6f0d287eaf","Type":"ContainerDied","Data":"0da1af733c106e8aa2bd0db13a67b82374ffa968846291e2afa32aa1e37cf3f4"} Oct 30 12:30:48.622563 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:48.622002 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46" event={"ID":"d11d3c04-4239-41d4-a099-3c6f0d287eaf","Type":"ContainerDied","Data":"ce552fe8f01287d9f56ab9eef9f59ef13d2ed41ee84eea1186eaed06fb875664"} Oct 30 12:30:48.622563 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:48.622037 4468 scope.go:122] "RemoveContainer" containerID="0da1af733c106e8aa2bd0db13a67b82374ffa968846291e2afa32aa1e37cf3f4" Oct 30 12:30:48.640644 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:48.640047 4468 scope.go:122] "RemoveContainer" containerID="0da1af733c106e8aa2bd0db13a67b82374ffa968846291e2afa32aa1e37cf3f4" Oct 30 12:30:48.640644 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:48.640557 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da1af733c106e8aa2bd0db13a67b82374ffa968846291e2afa32aa1e37cf3f4\": container with ID starting with 0da1af733c106e8aa2bd0db13a67b82374ffa968846291e2afa32aa1e37cf3f4 not found: ID does not exist" containerID="0da1af733c106e8aa2bd0db13a67b82374ffa968846291e2afa32aa1e37cf3f4" Oct 30 12:30:48.640644 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:48.640594 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da1af733c106e8aa2bd0db13a67b82374ffa968846291e2afa32aa1e37cf3f4"} err="failed to get container status \"0da1af733c106e8aa2bd0db13a67b82374ffa968846291e2afa32aa1e37cf3f4\": rpc error: code = NotFound desc = could not find container \"0da1af733c106e8aa2bd0db13a67b82374ffa968846291e2afa32aa1e37cf3f4\": container with ID starting with 0da1af733c106e8aa2bd0db13a67b82374ffa968846291e2afa32aa1e37cf3f4 not found: ID does not exist" Oct 30 12:30:48.658708 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:48.658678 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rhws\" (UniqueName: \"kubernetes.io/projected/d11d3c04-4239-41d4-a099-3c6f0d287eaf-kube-api-access-9rhws\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:48.658708 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:48.658708 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11d3c04-4239-41d4-a099-3c6f0d287eaf-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:48.682709 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:48.682664 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46"] Oct 30 12:30:48.689326 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:48.689299 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7eee3b-664d78c5d4-67f46"] Oct 30 12:30:49.090529 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:49.090492 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d11d3c04-4239-41d4-a099-3c6f0d287eaf" path="/var/lib/kubelet/pods/d11d3c04-4239-41d4-a099-3c6f0d287eaf/volumes" Oct 30 12:30:49.265816 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:49.265710 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg"] Oct 30 12:30:51.594799 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:51.594725 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" Oct 30 12:30:53.600101 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:53.600042 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:30:54.266409 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:54.266371 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg" podUID="8a160a20-d339-40f4-9c7a-8457342868d3" containerName="payload-pre-processing" containerID="cri-o://4da132724c390b2194583fd1bf72e34ab13cf276364cc7037430678f509ea4a1" gracePeriod=25 Oct 30 12:30:54.604157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:54.604129 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg" Oct 30 12:30:54.672020 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:54.671985 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a160a20-d339-40f4-9c7a-8457342868d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a160a20-d339-40f4-9c7a-8457342868d3-plugins-config-volume\") pod \"8a160a20-d339-40f4-9c7a-8457342868d3\" (UID: \"8a160a20-d339-40f4-9c7a-8457342868d3\") " Oct 30 12:30:54.672256 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:54.672233 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a160a20-d339-40f4-9c7a-8457342868d3-kube-api-access-z6tlr\" (UniqueName: \"kubernetes.io/projected/8a160a20-d339-40f4-9c7a-8457342868d3-kube-api-access-z6tlr\") pod \"8a160a20-d339-40f4-9c7a-8457342868d3\" (UID: \"8a160a20-d339-40f4-9c7a-8457342868d3\") " Oct 30 12:30:54.672419 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:54.672396 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a160a20-d339-40f4-9c7a-8457342868d3-plugins-config-volume" pod "8a160a20-d339-40f4-9c7a-8457342868d3" (UID: "8a160a20-d339-40f4-9c7a-8457342868d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:54.672787 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:54.672762 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a160a20-d339-40f4-9c7a-8457342868d3-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:54.675171 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:54.675119 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a160a20-d339-40f4-9c7a-8457342868d3-kube-api-access-z6tlr" pod "8a160a20-d339-40f4-9c7a-8457342868d3" (UID: "8a160a20-d339-40f4-9c7a-8457342868d3"). InnerVolumeSpecName "kube-api-access-z6tlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:54.679864 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:54.679839 4468 generic.go:356] "Generic (PLEG): container finished" podID="8a160a20-d339-40f4-9c7a-8457342868d3" containerID="4da132724c390b2194583fd1bf72e34ab13cf276364cc7037430678f509ea4a1" exitCode=0 Oct 30 12:30:54.679979 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:54.679888 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg" Oct 30 12:30:54.679979 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:54.679925 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg" event={"ID":"8a160a20-d339-40f4-9c7a-8457342868d3","Type":"ContainerDied","Data":"4da132724c390b2194583fd1bf72e34ab13cf276364cc7037430678f509ea4a1"} Oct 30 12:30:54.680141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:54.679991 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg" event={"ID":"8a160a20-d339-40f4-9c7a-8457342868d3","Type":"ContainerDied","Data":"ccfa046c81a6d50d79eaf2b911cd9ea1ee84d5357d327591305c7cfaa1c97e82"} Oct 30 12:30:54.680141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:54.680027 4468 scope.go:122] "RemoveContainer" containerID="4da132724c390b2194583fd1bf72e34ab13cf276364cc7037430678f509ea4a1" Oct 30 12:30:54.700353 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:54.700332 4468 scope.go:122] "RemoveContainer" containerID="4da132724c390b2194583fd1bf72e34ab13cf276364cc7037430678f509ea4a1" Oct 30 12:30:54.700688 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:54.700668 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4da132724c390b2194583fd1bf72e34ab13cf276364cc7037430678f509ea4a1\": container with ID starting with 4da132724c390b2194583fd1bf72e34ab13cf276364cc7037430678f509ea4a1 not found: ID does not exist" containerID="4da132724c390b2194583fd1bf72e34ab13cf276364cc7037430678f509ea4a1" Oct 30 12:30:54.700760 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:54.700706 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da132724c390b2194583fd1bf72e34ab13cf276364cc7037430678f509ea4a1"} err="failed to get container status \"4da132724c390b2194583fd1bf72e34ab13cf276364cc7037430678f509ea4a1\": rpc error: code = NotFound desc = could not find container \"4da132724c390b2194583fd1bf72e34ab13cf276364cc7037430678f509ea4a1\": container with ID starting with 4da132724c390b2194583fd1bf72e34ab13cf276364cc7037430678f509ea4a1 not found: ID does not exist" Oct 30 12:30:54.746842 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:54.746809 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg"] Oct 30 12:30:54.762274 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:54.762237 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ec236e-7bf78d8959-6b6zg"] Oct 30 12:30:54.773719 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:54.773692 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6tlr\" (UniqueName: \"kubernetes.io/projected/8a160a20-d339-40f4-9c7a-8457342868d3-kube-api-access-z6tlr\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:55.090636 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:55.090600 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a160a20-d339-40f4-9c7a-8457342868d3" path="/var/lib/kubelet/pods/8a160a20-d339-40f4-9c7a-8457342868d3/volumes" Oct 30 12:30:56.378972 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:56.378932 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r"] Oct 30 12:30:56.379488 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:56.379257 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" podUID="3acebec4-84f8-415a-84f7-f8f7d9391a85" containerName="istio-proxy" containerID="cri-o://7971141013ff3fff8cb233b410257fde691a6ba8ec23dc0620567c64f830b2eb" gracePeriod=30 Oct 30 12:30:57.963185 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:57.963145 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" podUID="17be3a09-0989-4ccb-ae91-4a316e0fdca8" containerName="main" containerID="cri-o://47ead5edb7c82b5ed3641c1e132e7e366a5022f7c7e151d5b3f7cdaa8b0bc6d5" gracePeriod=45 Oct 30 12:30:58.364994 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.364967 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:30:58.502766 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.502695 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-model-cache\") pod \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " Oct 30 12:30:58.502912 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.502770 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17be3a09-0989-4ccb-ae91-4a316e0fdca8-tls-certs\" (UniqueName: \"kubernetes.io/secret/17be3a09-0989-4ccb-ae91-4a316e0fdca8-tls-certs\") pod \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " Oct 30 12:30:58.502963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.502916 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-tmp-dir\") pod \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " Oct 30 12:30:58.503003 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.502974 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-model-cache" pod "17be3a09-0989-4ccb-ae91-4a316e0fdca8" (UID: "17be3a09-0989-4ccb-ae91-4a316e0fdca8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:58.503062 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.503040 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17be3a09-0989-4ccb-ae91-4a316e0fdca8-kube-api-access-2j6bj\" (UniqueName: \"kubernetes.io/projected/17be3a09-0989-4ccb-ae91-4a316e0fdca8-kube-api-access-2j6bj\") pod \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " Oct 30 12:30:58.503145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.503119 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-dshm\") pod \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " Oct 30 12:30:58.503201 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.503164 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-home\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-home\") pod \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\" (UID: \"17be3a09-0989-4ccb-ae91-4a316e0fdca8\") " Oct 30 12:30:58.512092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.503669 4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-model-cache\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:58.512315 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.512141 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-tmp-dir" pod "17be3a09-0989-4ccb-ae91-4a316e0fdca8" (UID: "17be3a09-0989-4ccb-ae91-4a316e0fdca8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:58.532609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.532179 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-home" pod "17be3a09-0989-4ccb-ae91-4a316e0fdca8" (UID: "17be3a09-0989-4ccb-ae91-4a316e0fdca8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:58.542140 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.535975 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-dshm" pod "17be3a09-0989-4ccb-ae91-4a316e0fdca8" (UID: "17be3a09-0989-4ccb-ae91-4a316e0fdca8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:58.544043 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.544005 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17be3a09-0989-4ccb-ae91-4a316e0fdca8-tls-certs" pod "17be3a09-0989-4ccb-ae91-4a316e0fdca8" (UID: "17be3a09-0989-4ccb-ae91-4a316e0fdca8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:30:58.544365 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.544335 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17be3a09-0989-4ccb-ae91-4a316e0fdca8-kube-api-access-2j6bj" pod "17be3a09-0989-4ccb-ae91-4a316e0fdca8" (UID: "17be3a09-0989-4ccb-ae91-4a316e0fdca8"). InnerVolumeSpecName "kube-api-access-2j6bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:58.565943 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.565915 4468 patch_prober.go:29] interesting pod/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:30:58.566046 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.565984 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" podUID="3acebec4-84f8-415a-84f7-f8f7d9391a85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:30:58.626799 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.626687 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j6bj\" (UniqueName: \"kubernetes.io/projected/17be3a09-0989-4ccb-ae91-4a316e0fdca8-kube-api-access-2j6bj\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:58.626799 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.626727 4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-dshm\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:58.626799 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.626744 4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-home\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:58.626799 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.626758 4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17be3a09-0989-4ccb-ae91-4a316e0fdca8-tls-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:58.626799 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.626772 4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17be3a09-0989-4ccb-ae91-4a316e0fdca8-tmp-dir\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:30:58.732621 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.732579 4468 generic.go:356] "Generic (PLEG): container finished" podID="17be3a09-0989-4ccb-ae91-4a316e0fdca8" containerID="47ead5edb7c82b5ed3641c1e132e7e366a5022f7c7e151d5b3f7cdaa8b0bc6d5" exitCode=0 Oct 30 12:30:58.732847 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.732623 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" Oct 30 12:30:58.732847 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.732654 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" event={"ID":"17be3a09-0989-4ccb-ae91-4a316e0fdca8","Type":"ContainerDied","Data":"47ead5edb7c82b5ed3641c1e132e7e366a5022f7c7e151d5b3f7cdaa8b0bc6d5"} Oct 30 12:30:58.732847 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.732698 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj" event={"ID":"17be3a09-0989-4ccb-ae91-4a316e0fdca8","Type":"ContainerDied","Data":"50801ba96b679ff6ea420a3c2b146d4faea5ed1b8357300703eed9dcb1f6a02d"} Oct 30 12:30:58.732847 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.732718 4468 scope.go:122] "RemoveContainer" containerID="47ead5edb7c82b5ed3641c1e132e7e366a5022f7c7e151d5b3f7cdaa8b0bc6d5" Oct 30 12:30:58.743133 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.743106 4468 scope.go:122] "RemoveContainer" containerID="47ead5edb7c82b5ed3641c1e132e7e366a5022f7c7e151d5b3f7cdaa8b0bc6d5" Oct 30 12:30:58.743404 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:30:58.743376 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ead5edb7c82b5ed3641c1e132e7e366a5022f7c7e151d5b3f7cdaa8b0bc6d5\": container with ID starting with 47ead5edb7c82b5ed3641c1e132e7e366a5022f7c7e151d5b3f7cdaa8b0bc6d5 not found: ID does not exist" containerID="47ead5edb7c82b5ed3641c1e132e7e366a5022f7c7e151d5b3f7cdaa8b0bc6d5" Oct 30 12:30:58.743475 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.743405 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ead5edb7c82b5ed3641c1e132e7e366a5022f7c7e151d5b3f7cdaa8b0bc6d5"} err="failed to get container status \"47ead5edb7c82b5ed3641c1e132e7e366a5022f7c7e151d5b3f7cdaa8b0bc6d5\": rpc error: code = NotFound desc = could not find container \"47ead5edb7c82b5ed3641c1e132e7e366a5022f7c7e151d5b3f7cdaa8b0bc6d5\": container with ID starting with 47ead5edb7c82b5ed3641c1e132e7e366a5022f7c7e151d5b3f7cdaa8b0bc6d5 not found: ID does not exist" Oct 30 12:30:58.790986 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.790908 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj"] Oct 30 12:30:58.795657 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:58.795633 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-8e3112/ipp-route-8e3112-kserve-7c789474df-fnfmj"] Oct 30 12:30:59.091905 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:30:59.091832 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17be3a09-0989-4ccb-ae91-4a316e0fdca8" path="/var/lib/kubelet/pods/17be3a09-0989-4ccb-ae91-4a316e0fdca8/volumes" Oct 30 12:31:01.076179 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.076143 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p"] Oct 30 12:31:01.076666 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.076375 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" podUID="f1d20bcb-72ac-4692-9f8e-a064a2886bce" containerName="maas-api" containerID="cri-o://5664ccc038772a6c801f14306556ef2b7f64a37585a0f13633fd5b69b5b43445" gracePeriod=30 Oct 30 12:31:01.681807 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.681776 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" Oct 30 12:31:01.689656 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.689086 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d20bcb-72ac-4692-9f8e-a064a2886bce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d20bcb-72ac-4692-9f8e-a064a2886bce-metrics-certs\") pod \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\" (UID: \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\") " Oct 30 12:31:01.689656 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.689121 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d20bcb-72ac-4692-9f8e-a064a2886bce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d20bcb-72ac-4692-9f8e-a064a2886bce-maas-api-tls\") pod \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\" (UID: \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\") " Oct 30 12:31:01.689656 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.689229 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d20bcb-72ac-4692-9f8e-a064a2886bce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d20bcb-72ac-4692-9f8e-a064a2886bce-openshift-service-ca\") pod \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\" (UID: \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\") " Oct 30 12:31:01.689656 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.689248 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d20bcb-72ac-4692-9f8e-a064a2886bce-kube-api-access-zrmnd\" (UniqueName: \"kubernetes.io/projected/f1d20bcb-72ac-4692-9f8e-a064a2886bce-kube-api-access-zrmnd\") pod \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\" (UID: \"f1d20bcb-72ac-4692-9f8e-a064a2886bce\") " Oct 30 12:31:01.689967 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.689941 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d20bcb-72ac-4692-9f8e-a064a2886bce-openshift-service-ca" pod "f1d20bcb-72ac-4692-9f8e-a064a2886bce" (UID: "f1d20bcb-72ac-4692-9f8e-a064a2886bce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:01.691725 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.691702 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d20bcb-72ac-4692-9f8e-a064a2886bce-metrics-certs" pod "f1d20bcb-72ac-4692-9f8e-a064a2886bce" (UID: "f1d20bcb-72ac-4692-9f8e-a064a2886bce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:31:01.692130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.692111 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d20bcb-72ac-4692-9f8e-a064a2886bce-kube-api-access-zrmnd" pod "f1d20bcb-72ac-4692-9f8e-a064a2886bce" (UID: "f1d20bcb-72ac-4692-9f8e-a064a2886bce"). InnerVolumeSpecName "kube-api-access-zrmnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:01.692212 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.692144 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d20bcb-72ac-4692-9f8e-a064a2886bce-maas-api-tls" pod "f1d20bcb-72ac-4692-9f8e-a064a2886bce" (UID: "f1d20bcb-72ac-4692-9f8e-a064a2886bce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:31:01.757736 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.757688 4468 generic.go:356] "Generic (PLEG): container finished" podID="f1d20bcb-72ac-4692-9f8e-a064a2886bce" containerID="5664ccc038772a6c801f14306556ef2b7f64a37585a0f13633fd5b69b5b43445" exitCode=0 Oct 30 12:31:01.757923 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.757804 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" event={"ID":"f1d20bcb-72ac-4692-9f8e-a064a2886bce","Type":"ContainerDied","Data":"5664ccc038772a6c801f14306556ef2b7f64a37585a0f13633fd5b69b5b43445"} Oct 30 12:31:01.757923 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.757874 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" event={"ID":"f1d20bcb-72ac-4692-9f8e-a064a2886bce","Type":"ContainerDied","Data":"ad3ccce2b2485bd62e77b04cd0227a57cf7ea638fe88fc445418dc2f1a260508"} Oct 30 12:31:01.758038 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.757923 4468 scope.go:122] "RemoveContainer" containerID="5664ccc038772a6c801f14306556ef2b7f64a37585a0f13633fd5b69b5b43445" Oct 30 12:31:01.758268 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.758247 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p" Oct 30 12:31:01.764957 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.764927 4468 generic.go:356] "Generic (PLEG): container finished" podID="3acebec4-84f8-415a-84f7-f8f7d9391a85" containerID="7971141013ff3fff8cb233b410257fde691a6ba8ec23dc0620567c64f830b2eb" exitCode=0 Oct 30 12:31:01.765215 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.764999 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" event={"ID":"3acebec4-84f8-415a-84f7-f8f7d9391a85","Type":"ContainerDied","Data":"7971141013ff3fff8cb233b410257fde691a6ba8ec23dc0620567c64f830b2eb"} Oct 30 12:31:01.776260 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.776241 4468 scope.go:122] "RemoveContainer" containerID="5664ccc038772a6c801f14306556ef2b7f64a37585a0f13633fd5b69b5b43445" Oct 30 12:31:01.776558 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:31:01.776539 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5664ccc038772a6c801f14306556ef2b7f64a37585a0f13633fd5b69b5b43445\": container with ID starting with 5664ccc038772a6c801f14306556ef2b7f64a37585a0f13633fd5b69b5b43445 not found: ID does not exist" containerID="5664ccc038772a6c801f14306556ef2b7f64a37585a0f13633fd5b69b5b43445" Oct 30 12:31:01.776599 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.776569 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5664ccc038772a6c801f14306556ef2b7f64a37585a0f13633fd5b69b5b43445"} err="failed to get container status \"5664ccc038772a6c801f14306556ef2b7f64a37585a0f13633fd5b69b5b43445\": rpc error: code = NotFound desc = could not find container \"5664ccc038772a6c801f14306556ef2b7f64a37585a0f13633fd5b69b5b43445\": container with ID starting with 5664ccc038772a6c801f14306556ef2b7f64a37585a0f13633fd5b69b5b43445 not found: ID does not exist" Oct 30 12:31:01.776599 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.776586 4468 scope.go:122] "RemoveContainer" containerID="8013cb7d7a25a3866bb4699d4e861a05650bbb4e55c1fcffac5d0b78b20174e9" Oct 30 12:31:01.790727 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.790682 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d20bcb-72ac-4692-9f8e-a064a2886bce-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:01.790727 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.790714 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d20bcb-72ac-4692-9f8e-a064a2886bce-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:01.790727 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.790729 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d20bcb-72ac-4692-9f8e-a064a2886bce-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:01.791119 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.790744 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrmnd\" (UniqueName: \"kubernetes.io/projected/f1d20bcb-72ac-4692-9f8e-a064a2886bce-kube-api-access-zrmnd\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:01.849190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.848822 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p"] Oct 30 12:31:01.854835 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.854797 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec236e-fb4b559cb-mg87p"] Oct 30 12:31:01.884112 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.884080 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:31:01.892181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.892143 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-workload-certs\") pod \"3acebec4-84f8-415a-84f7-f8f7d9391a85\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " Oct 30 12:31:01.892343 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.892212 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-workload-socket\") pod \"3acebec4-84f8-415a-84f7-f8f7d9391a85\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " Oct 30 12:31:01.892343 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.892238 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-credential-socket\") pod \"3acebec4-84f8-415a-84f7-f8f7d9391a85\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " Oct 30 12:31:01.892343 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.892271 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-token\" (UniqueName: \"kubernetes.io/projected/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-token\") pod \"3acebec4-84f8-415a-84f7-f8f7d9391a85\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " Oct 30 12:31:01.892603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.892357 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3acebec4-84f8-415a-84f7-f8f7d9391a85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3acebec4-84f8-415a-84f7-f8f7d9391a85-istiod-ca-cert\") pod \"3acebec4-84f8-415a-84f7-f8f7d9391a85\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " Oct 30 12:31:01.892603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.892384 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-workload-certs" pod "3acebec4-84f8-415a-84f7-f8f7d9391a85" (UID: "3acebec4-84f8-415a-84f7-f8f7d9391a85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:01.892603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.892451 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-podinfo\") pod \"3acebec4-84f8-415a-84f7-f8f7d9391a85\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " Oct 30 12:31:01.892603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.892492 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-data\") pod \"3acebec4-84f8-415a-84f7-f8f7d9391a85\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " Oct 30 12:31:01.892603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.892523 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-workload-socket" pod "3acebec4-84f8-415a-84f7-f8f7d9391a85" (UID: "3acebec4-84f8-415a-84f7-f8f7d9391a85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:01.892603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.892544 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-credential-socket" pod "3acebec4-84f8-415a-84f7-f8f7d9391a85" (UID: "3acebec4-84f8-415a-84f7-f8f7d9391a85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:01.892877 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.892629 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-envoy\") pod \"3acebec4-84f8-415a-84f7-f8f7d9391a85\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " Oct 30 12:31:01.892877 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.892695 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3acebec4-84f8-415a-84f7-f8f7d9391a85-kube-api-access-fwkf4\" (UniqueName: \"kubernetes.io/projected/3acebec4-84f8-415a-84f7-f8f7d9391a85-kube-api-access-fwkf4\") pod \"3acebec4-84f8-415a-84f7-f8f7d9391a85\" (UID: \"3acebec4-84f8-415a-84f7-f8f7d9391a85\") " Oct 30 12:31:01.892877 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.892699 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3acebec4-84f8-415a-84f7-f8f7d9391a85-istiod-ca-cert" pod "3acebec4-84f8-415a-84f7-f8f7d9391a85" (UID: "3acebec4-84f8-415a-84f7-f8f7d9391a85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:01.892877 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.892801 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-data" pod "3acebec4-84f8-415a-84f7-f8f7d9391a85" (UID: "3acebec4-84f8-415a-84f7-f8f7d9391a85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:01.893183 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.893166 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3acebec4-84f8-415a-84f7-f8f7d9391a85-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:01.893183 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.893183 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:01.893313 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.893192 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:01.893313 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.893200 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:01.893313 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.893211 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:01.894625 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.894601 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-token" pod "3acebec4-84f8-415a-84f7-f8f7d9391a85" (UID: "3acebec4-84f8-415a-84f7-f8f7d9391a85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:01.894857 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.894840 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acebec4-84f8-415a-84f7-f8f7d9391a85-kube-api-access-fwkf4" pod "3acebec4-84f8-415a-84f7-f8f7d9391a85" (UID: "3acebec4-84f8-415a-84f7-f8f7d9391a85"). InnerVolumeSpecName "kube-api-access-fwkf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:01.895106 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.895063 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-envoy" pod "3acebec4-84f8-415a-84f7-f8f7d9391a85" (UID: "3acebec4-84f8-415a-84f7-f8f7d9391a85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:01.895166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:01.895087 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-podinfo" pod "3acebec4-84f8-415a-84f7-f8f7d9391a85" (UID: "3acebec4-84f8-415a-84f7-f8f7d9391a85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:31:02.043148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:02.042389 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:02.043148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:02.042444 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwkf4\" (UniqueName: \"kubernetes.io/projected/3acebec4-84f8-415a-84f7-f8f7d9391a85-kube-api-access-fwkf4\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:02.043148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:02.042463 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:02.043148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:02.042491 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3acebec4-84f8-415a-84f7-f8f7d9391a85-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:02.774002 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:02.773975 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" Oct 30 12:31:02.774517 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:02.773999 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r" event={"ID":"3acebec4-84f8-415a-84f7-f8f7d9391a85","Type":"ContainerDied","Data":"5d4e2c3162de31923a52447165bcc0966189ca9bbf2cfc9f14fc214b80c3bd06"} Oct 30 12:31:02.774517 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:02.774135 4468 scope.go:122] "RemoveContainer" containerID="7971141013ff3fff8cb233b410257fde691a6ba8ec23dc0620567c64f830b2eb" Oct 30 12:31:02.868013 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:02.867982 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r"] Oct 30 12:31:02.881633 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:02.881600 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7eee3b-openshift-default-96456c9fc-dg27r"] Oct 30 12:31:03.089999 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:03.089919 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3acebec4-84f8-415a-84f7-f8f7d9391a85" path="/var/lib/kubelet/pods/3acebec4-84f8-415a-84f7-f8f7d9391a85/volumes" Oct 30 12:31:03.090447 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:03.090431 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d20bcb-72ac-4692-9f8e-a064a2886bce" path="/var/lib/kubelet/pods/f1d20bcb-72ac-4692-9f8e-a064a2886bce/volumes" Oct 30 12:31:07.818391 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:07.818356 4468 generic.go:356] "Generic (PLEG): container finished" podID="3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" containerID="9945fe7249d029828d5e8c11ddd1b29b6106cd08c4e3e88318fdf5d6dfa59ca1" exitCode=0 Oct 30 12:31:07.818945 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:07.818433 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" event={"ID":"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4","Type":"ContainerDied","Data":"9945fe7249d029828d5e8c11ddd1b29b6106cd08c4e3e88318fdf5d6dfa59ca1"} Oct 30 12:31:07.818945 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:07.818843 4468 scope.go:122] "RemoveContainer" containerID="9945fe7249d029828d5e8c11ddd1b29b6106cd08c4e3e88318fdf5d6dfa59ca1" Oct 30 12:31:07.820494 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:07.820461 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:31:07.820584 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:07.820535 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:31:07.820584 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:07.820569 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:31:08.834396 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:08.834359 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" event={"ID":"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4","Type":"ContainerStarted","Data":"774d11bb66ccc0cdb68ca19ad81c2a7e1e3cf199eb559e4fbc3be1e9f7d0cb71"} Oct 30 12:31:09.494250 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:09.494162 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:31:09.494568 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:09.494544 4468 patch_prober.go:29] interesting pod/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 12:31:09.494719 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:09.494695 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" podUID="3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 12:31:10.498511 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:10.498477 4468 patch_prober.go:29] interesting pod/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 12:31:10.498862 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:10.498537 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" podUID="3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 12:31:11.492800 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:11.492761 4468 patch_prober.go:29] interesting pod/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 12:31:11.492988 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:11.492821 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" podUID="3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 12:31:12.492312 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:12.492271 4468 patch_prober.go:29] interesting pod/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 12:31:12.492845 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:12.492356 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" podUID="3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 12:31:13.491972 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:13.491937 4468 patch_prober.go:29] interesting pod/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 12:31:13.492207 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:13.492006 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" podUID="3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 12:31:14.736895 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:14.736865 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:31:14.737425 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:14.737312 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:31:14.738712 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:14.738687 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:31:21.893529 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:21.893491 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs"] Oct 30 12:31:21.894046 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:21.893790 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" podUID="0166c51a-8d66-47e5-ba9a-2fb14b342215" containerName="istio-proxy" containerID="cri-o://e59b16c0ba13ce257c13c780a8c300f573a6b83394a53273661b52c441d8f0c3" gracePeriod=30 Oct 30 12:31:23.598689 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:31:23.598650 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:31:27.146672 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.146207 4468 generic.go:356] "Generic (PLEG): container finished" podID="0166c51a-8d66-47e5-ba9a-2fb14b342215" containerID="e59b16c0ba13ce257c13c780a8c300f573a6b83394a53273661b52c441d8f0c3" exitCode=0 Oct 30 12:31:27.146672 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.146293 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" event={"ID":"0166c51a-8d66-47e5-ba9a-2fb14b342215","Type":"ContainerDied","Data":"e59b16c0ba13ce257c13c780a8c300f573a6b83394a53273661b52c441d8f0c3"} Oct 30 12:31:27.146672 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.146352 4468 scope.go:122] "RemoveContainer" containerID="63d2a29bfbed3a8d0e2aad6a1b5d57f37cf82702804e1b3638fda411f6f17eb4" Oct 30 12:31:27.175211 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:31:27.175164 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d2a29bfbed3a8d0e2aad6a1b5d57f37cf82702804e1b3638fda411f6f17eb4\": container with ID starting with 63d2a29bfbed3a8d0e2aad6a1b5d57f37cf82702804e1b3638fda411f6f17eb4 not found: ID does not exist" containerID="63d2a29bfbed3a8d0e2aad6a1b5d57f37cf82702804e1b3638fda411f6f17eb4" Oct 30 12:31:27.176792 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.176765 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:31:27.247278 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.247248 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-workload-certs\") pod \"0166c51a-8d66-47e5-ba9a-2fb14b342215\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " Oct 30 12:31:27.247459 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.247293 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-envoy\") pod \"0166c51a-8d66-47e5-ba9a-2fb14b342215\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " Oct 30 12:31:27.247459 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.247353 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-workload-socket\") pod \"0166c51a-8d66-47e5-ba9a-2fb14b342215\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " Oct 30 12:31:27.247459 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.247396 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-token\" (UniqueName: \"kubernetes.io/projected/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-token\") pod \"0166c51a-8d66-47e5-ba9a-2fb14b342215\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " Oct 30 12:31:27.247585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.247516 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-credential-socket\") pod \"0166c51a-8d66-47e5-ba9a-2fb14b342215\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " Oct 30 12:31:27.247585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.247566 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0166c51a-8d66-47e5-ba9a-2fb14b342215-kube-api-access-qpl8b\" (UniqueName: \"kubernetes.io/projected/0166c51a-8d66-47e5-ba9a-2fb14b342215-kube-api-access-qpl8b\") pod \"0166c51a-8d66-47e5-ba9a-2fb14b342215\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " Oct 30 12:31:27.247670 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.247612 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0166c51a-8d66-47e5-ba9a-2fb14b342215-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0166c51a-8d66-47e5-ba9a-2fb14b342215-istiod-ca-cert\") pod \"0166c51a-8d66-47e5-ba9a-2fb14b342215\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " Oct 30 12:31:27.247725 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.247601 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-workload-certs" pod "0166c51a-8d66-47e5-ba9a-2fb14b342215" (UID: "0166c51a-8d66-47e5-ba9a-2fb14b342215"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:27.247725 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.247677 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-podinfo\") pod \"0166c51a-8d66-47e5-ba9a-2fb14b342215\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " Oct 30 12:31:27.247830 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.247767 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-data\") pod \"0166c51a-8d66-47e5-ba9a-2fb14b342215\" (UID: \"0166c51a-8d66-47e5-ba9a-2fb14b342215\") " Oct 30 12:31:27.247931 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.247895 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-workload-socket" pod "0166c51a-8d66-47e5-ba9a-2fb14b342215" (UID: "0166c51a-8d66-47e5-ba9a-2fb14b342215"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:27.248027 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.248005 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0166c51a-8d66-47e5-ba9a-2fb14b342215-istiod-ca-cert" pod "0166c51a-8d66-47e5-ba9a-2fb14b342215" (UID: "0166c51a-8d66-47e5-ba9a-2fb14b342215"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:27.248372 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.248347 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-data" pod "0166c51a-8d66-47e5-ba9a-2fb14b342215" (UID: "0166c51a-8d66-47e5-ba9a-2fb14b342215"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:27.248478 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.248387 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-credential-socket" pod "0166c51a-8d66-47e5-ba9a-2fb14b342215" (UID: "0166c51a-8d66-47e5-ba9a-2fb14b342215"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:27.248478 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.248424 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0166c51a-8d66-47e5-ba9a-2fb14b342215-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:27.248478 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.248445 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:27.248478 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.248460 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:27.250036 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.250009 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-podinfo" pod "0166c51a-8d66-47e5-ba9a-2fb14b342215" (UID: "0166c51a-8d66-47e5-ba9a-2fb14b342215"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:31:27.250726 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.250614 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0166c51a-8d66-47e5-ba9a-2fb14b342215-kube-api-access-qpl8b" pod "0166c51a-8d66-47e5-ba9a-2fb14b342215" (UID: "0166c51a-8d66-47e5-ba9a-2fb14b342215"). InnerVolumeSpecName "kube-api-access-qpl8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:27.250828 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.250805 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-token" pod "0166c51a-8d66-47e5-ba9a-2fb14b342215" (UID: "0166c51a-8d66-47e5-ba9a-2fb14b342215"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:27.252322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.252284 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-envoy" pod "0166c51a-8d66-47e5-ba9a-2fb14b342215" (UID: "0166c51a-8d66-47e5-ba9a-2fb14b342215"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:27.349423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.349387 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:27.349423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.349420 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:27.349423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.349430 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpl8b\" (UniqueName: \"kubernetes.io/projected/0166c51a-8d66-47e5-ba9a-2fb14b342215-kube-api-access-qpl8b\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:27.349661 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.349439 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:27.349661 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.349449 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:27.349661 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:27.349456 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0166c51a-8d66-47e5-ba9a-2fb14b342215-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:28.154027 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:28.153994 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" Oct 30 12:31:28.154027 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:28.154016 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs" event={"ID":"0166c51a-8d66-47e5-ba9a-2fb14b342215","Type":"ContainerDied","Data":"3d10a592fb143c71f3863b73889f379e6b9743435a55e5bae358fc2dded20d5e"} Oct 30 12:31:28.154528 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:28.154098 4468 scope.go:122] "RemoveContainer" containerID="e59b16c0ba13ce257c13c780a8c300f573a6b83394a53273661b52c441d8f0c3" Oct 30 12:31:28.227004 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:28.226962 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs"] Oct 30 12:31:28.240190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:28.240158 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-41bdda-openshift-default-7cc4dd8b5-4csjs"] Oct 30 12:31:29.093755 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:29.093722 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0166c51a-8d66-47e5-ba9a-2fb14b342215" path="/var/lib/kubelet/pods/0166c51a-8d66-47e5-ba9a-2fb14b342215/volumes" Oct 30 12:31:33.023411 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:33.023371 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp"] Oct 30 12:31:38.023881 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:38.023821 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp" podUID="42e6150b-6f32-4d92-911c-962f73ba0e68" containerName="payload-pre-processing" containerID="cri-o://652eaa215dc8f821031626c40a207b5aa4077ef5e7e5771d0cbaf512db68e65a" gracePeriod=25 Oct 30 12:31:38.277645 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:38.277555 4468 generic.go:356] "Generic (PLEG): container finished" podID="42e6150b-6f32-4d92-911c-962f73ba0e68" containerID="652eaa215dc8f821031626c40a207b5aa4077ef5e7e5771d0cbaf512db68e65a" exitCode=0 Oct 30 12:31:38.278060 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:38.278031 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp" event={"ID":"42e6150b-6f32-4d92-911c-962f73ba0e68","Type":"ContainerDied","Data":"652eaa215dc8f821031626c40a207b5aa4077ef5e7e5771d0cbaf512db68e65a"} Oct 30 12:31:38.688645 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:38.688596 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp" Oct 30 12:31:38.853091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:38.852620 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42e6150b-6f32-4d92-911c-962f73ba0e68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e6150b-6f32-4d92-911c-962f73ba0e68-plugins-config-volume\") pod \"42e6150b-6f32-4d92-911c-962f73ba0e68\" (UID: \"42e6150b-6f32-4d92-911c-962f73ba0e68\") " Oct 30 12:31:38.853091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:38.852685 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42e6150b-6f32-4d92-911c-962f73ba0e68-kube-api-access-jfd7g\" (UniqueName: \"kubernetes.io/projected/42e6150b-6f32-4d92-911c-962f73ba0e68-kube-api-access-jfd7g\") pod \"42e6150b-6f32-4d92-911c-962f73ba0e68\" (UID: \"42e6150b-6f32-4d92-911c-962f73ba0e68\") " Oct 30 12:31:38.873094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:38.869680 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42e6150b-6f32-4d92-911c-962f73ba0e68-plugins-config-volume" pod "42e6150b-6f32-4d92-911c-962f73ba0e68" (UID: "42e6150b-6f32-4d92-911c-962f73ba0e68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:38.905397 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:38.905346 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e6150b-6f32-4d92-911c-962f73ba0e68-kube-api-access-jfd7g" pod "42e6150b-6f32-4d92-911c-962f73ba0e68" (UID: "42e6150b-6f32-4d92-911c-962f73ba0e68"). InnerVolumeSpecName "kube-api-access-jfd7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:38.954620 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:38.954518 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e6150b-6f32-4d92-911c-962f73ba0e68-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:38.954849 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:38.954831 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfd7g\" (UniqueName: \"kubernetes.io/projected/42e6150b-6f32-4d92-911c-962f73ba0e68-kube-api-access-jfd7g\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:39.286538 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:39.286436 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp" event={"ID":"42e6150b-6f32-4d92-911c-962f73ba0e68","Type":"ContainerDied","Data":"9129e880cadfe031fdcede0ab0ab82182094468dd3348a14f3bcd4a1f003a1ef"} Oct 30 12:31:39.286538 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:39.286461 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp" Oct 30 12:31:39.286538 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:39.286508 4468 scope.go:122] "RemoveContainer" containerID="652eaa215dc8f821031626c40a207b5aa4077ef5e7e5771d0cbaf512db68e65a" Oct 30 12:31:39.387812 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:39.387754 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp"] Oct 30 12:31:39.442169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:39.442126 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1e9675-56dc9fdd47-k5fcp"] Oct 30 12:31:41.120453 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:41.120413 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42e6150b-6f32-4d92-911c-962f73ba0e68" path="/var/lib/kubelet/pods/42e6150b-6f32-4d92-911c-962f73ba0e68/volumes" Oct 30 12:31:44.421205 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:44.421168 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k"] Oct 30 12:31:44.421664 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:44.421401 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" podUID="df475a56-3557-40c3-84bb-0e812127ae21" containerName="istio-proxy" containerID="cri-o://173bebb153208b2cc8c4908438b281747f4cee56d3f5fac2c677ff7a30b0e1ec" gracePeriod=30 Oct 30 12:31:49.698992 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.698921 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:31:49.744221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.744188 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df475a56-3557-40c3-84bb-0e812127ae21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df475a56-3557-40c3-84bb-0e812127ae21-istio-podinfo\") pod \"df475a56-3557-40c3-84bb-0e812127ae21\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " Oct 30 12:31:49.744401 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.744269 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-credential-socket\") pod \"df475a56-3557-40c3-84bb-0e812127ae21\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " Oct 30 12:31:49.744401 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.744341 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-istio-envoy\") pod \"df475a56-3557-40c3-84bb-0e812127ae21\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " Oct 30 12:31:49.744401 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.744374 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df475a56-3557-40c3-84bb-0e812127ae21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df475a56-3557-40c3-84bb-0e812127ae21-istiod-ca-cert\") pod \"df475a56-3557-40c3-84bb-0e812127ae21\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " Oct 30 12:31:49.744582 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.744405 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-istio-data\") pod \"df475a56-3557-40c3-84bb-0e812127ae21\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " Oct 30 12:31:49.744582 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.744438 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df475a56-3557-40c3-84bb-0e812127ae21-istio-token\" (UniqueName: \"kubernetes.io/projected/df475a56-3557-40c3-84bb-0e812127ae21-istio-token\") pod \"df475a56-3557-40c3-84bb-0e812127ae21\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " Oct 30 12:31:49.744582 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.744485 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-workload-socket\") pod \"df475a56-3557-40c3-84bb-0e812127ae21\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " Oct 30 12:31:49.744739 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.744610 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-credential-socket" pod "df475a56-3557-40c3-84bb-0e812127ae21" (UID: "df475a56-3557-40c3-84bb-0e812127ae21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:49.744799 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.744731 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df475a56-3557-40c3-84bb-0e812127ae21-kube-api-access-69snw\" (UniqueName: \"kubernetes.io/projected/df475a56-3557-40c3-84bb-0e812127ae21-kube-api-access-69snw\") pod \"df475a56-3557-40c3-84bb-0e812127ae21\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " Oct 30 12:31:49.744866 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.744803 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-workload-certs\") pod \"df475a56-3557-40c3-84bb-0e812127ae21\" (UID: \"df475a56-3557-40c3-84bb-0e812127ae21\") " Oct 30 12:31:49.744866 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.744821 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df475a56-3557-40c3-84bb-0e812127ae21-istiod-ca-cert" pod "df475a56-3557-40c3-84bb-0e812127ae21" (UID: "df475a56-3557-40c3-84bb-0e812127ae21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:49.745393 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.745206 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-workload-certs" pod "df475a56-3557-40c3-84bb-0e812127ae21" (UID: "df475a56-3557-40c3-84bb-0e812127ae21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:49.745393 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.745362 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-istio-data" pod "df475a56-3557-40c3-84bb-0e812127ae21" (UID: "df475a56-3557-40c3-84bb-0e812127ae21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:49.745585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.745481 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-workload-socket" pod "df475a56-3557-40c3-84bb-0e812127ae21" (UID: "df475a56-3557-40c3-84bb-0e812127ae21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:49.745585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.745562 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df475a56-3557-40c3-84bb-0e812127ae21-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:49.745585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.745585 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:49.745748 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.745599 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:49.745748 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.745618 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:49.747141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.747113 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-istio-envoy" pod "df475a56-3557-40c3-84bb-0e812127ae21" (UID: "df475a56-3557-40c3-84bb-0e812127ae21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:49.747412 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.747384 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df475a56-3557-40c3-84bb-0e812127ae21-kube-api-access-69snw" pod "df475a56-3557-40c3-84bb-0e812127ae21" (UID: "df475a56-3557-40c3-84bb-0e812127ae21"). InnerVolumeSpecName "kube-api-access-69snw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:49.747506 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.747448 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df475a56-3557-40c3-84bb-0e812127ae21-istio-podinfo" pod "df475a56-3557-40c3-84bb-0e812127ae21" (UID: "df475a56-3557-40c3-84bb-0e812127ae21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:31:49.747876 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.747855 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df475a56-3557-40c3-84bb-0e812127ae21-istio-token" pod "df475a56-3557-40c3-84bb-0e812127ae21" (UID: "df475a56-3557-40c3-84bb-0e812127ae21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:49.846710 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.846672 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69snw\" (UniqueName: \"kubernetes.io/projected/df475a56-3557-40c3-84bb-0e812127ae21-kube-api-access-69snw\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:49.846710 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.846702 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df475a56-3557-40c3-84bb-0e812127ae21-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:49.846710 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.846712 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:49.846943 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.846721 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df475a56-3557-40c3-84bb-0e812127ae21-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:49.846943 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:49.846729 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df475a56-3557-40c3-84bb-0e812127ae21-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:31:50.359147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:50.359107 4468 generic.go:356] "Generic (PLEG): container finished" podID="df475a56-3557-40c3-84bb-0e812127ae21" containerID="173bebb153208b2cc8c4908438b281747f4cee56d3f5fac2c677ff7a30b0e1ec" exitCode=0 Oct 30 12:31:50.359350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:50.359165 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" Oct 30 12:31:50.359350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:50.359184 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" event={"ID":"df475a56-3557-40c3-84bb-0e812127ae21","Type":"ContainerDied","Data":"173bebb153208b2cc8c4908438b281747f4cee56d3f5fac2c677ff7a30b0e1ec"} Oct 30 12:31:50.359350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:50.359242 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k" event={"ID":"df475a56-3557-40c3-84bb-0e812127ae21","Type":"ContainerDied","Data":"e4dbfb084f93df699b431b9bc545df6e714ff33ab10400b69746ba39dbf5d431"} Oct 30 12:31:50.359350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:50.359266 4468 scope.go:122] "RemoveContainer" containerID="173bebb153208b2cc8c4908438b281747f4cee56d3f5fac2c677ff7a30b0e1ec" Oct 30 12:31:50.370430 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:50.370402 4468 scope.go:122] "RemoveContainer" containerID="00f9b05fec387882161ab7e5a4de28ffd969d08326e5a0c690468e03a4d7a65a" Oct 30 12:31:50.379566 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:50.379545 4468 scope.go:122] "RemoveContainer" containerID="173bebb153208b2cc8c4908438b281747f4cee56d3f5fac2c677ff7a30b0e1ec" Oct 30 12:31:50.379828 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:31:50.379804 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"173bebb153208b2cc8c4908438b281747f4cee56d3f5fac2c677ff7a30b0e1ec\": container with ID starting with 173bebb153208b2cc8c4908438b281747f4cee56d3f5fac2c677ff7a30b0e1ec not found: ID does not exist" containerID="173bebb153208b2cc8c4908438b281747f4cee56d3f5fac2c677ff7a30b0e1ec" Oct 30 12:31:50.379905 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:50.379840 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"173bebb153208b2cc8c4908438b281747f4cee56d3f5fac2c677ff7a30b0e1ec"} err="failed to get container status \"173bebb153208b2cc8c4908438b281747f4cee56d3f5fac2c677ff7a30b0e1ec\": rpc error: code = NotFound desc = could not find container \"173bebb153208b2cc8c4908438b281747f4cee56d3f5fac2c677ff7a30b0e1ec\": container with ID starting with 173bebb153208b2cc8c4908438b281747f4cee56d3f5fac2c677ff7a30b0e1ec not found: ID does not exist" Oct 30 12:31:50.379905 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:50.379866 4468 scope.go:122] "RemoveContainer" containerID="00f9b05fec387882161ab7e5a4de28ffd969d08326e5a0c690468e03a4d7a65a" Oct 30 12:31:50.380153 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:31:50.380135 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00f9b05fec387882161ab7e5a4de28ffd969d08326e5a0c690468e03a4d7a65a\": container with ID starting with 00f9b05fec387882161ab7e5a4de28ffd969d08326e5a0c690468e03a4d7a65a not found: ID does not exist" containerID="00f9b05fec387882161ab7e5a4de28ffd969d08326e5a0c690468e03a4d7a65a" Oct 30 12:31:50.380206 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:50.380159 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00f9b05fec387882161ab7e5a4de28ffd969d08326e5a0c690468e03a4d7a65a"} err="failed to get container status \"00f9b05fec387882161ab7e5a4de28ffd969d08326e5a0c690468e03a4d7a65a\": rpc error: code = NotFound desc = could not find container \"00f9b05fec387882161ab7e5a4de28ffd969d08326e5a0c690468e03a4d7a65a\": container with ID starting with 00f9b05fec387882161ab7e5a4de28ffd969d08326e5a0c690468e03a4d7a65a not found: ID does not exist" Oct 30 12:31:50.435220 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:50.435188 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k"] Oct 30 12:31:50.449142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:50.449105 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-8e3112-openshift-default-6cf6458b87-6xw5k"] Oct 30 12:31:51.090371 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:31:51.090336 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df475a56-3557-40c3-84bb-0e812127ae21" path="/var/lib/kubelet/pods/df475a56-3557-40c3-84bb-0e812127ae21/volumes" Oct 30 12:31:53.602155 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:31:53.602113 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:32:07.826605 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:07.826562 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22"] Oct 30 12:32:07.827180 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:07.826790 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22" podUID="b6055421-f8de-4e98-bd0e-4f1a0cbeefc5" containerName="payload-processing" containerID="cri-o://5d0b9aa6e6b04c55b7144384b93da04556af9f01e31879f7110326a440d80dca" gracePeriod=30 Oct 30 12:32:08.095546 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:08.095479 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22" Oct 30 12:32:08.162812 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:08.162764 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6055421-f8de-4e98-bd0e-4f1a0cbeefc5-plugins-config-volume" pod "b6055421-f8de-4e98-bd0e-4f1a0cbeefc5" (UID: "b6055421-f8de-4e98-bd0e-4f1a0cbeefc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:08.164204 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:08.162264 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6055421-f8de-4e98-bd0e-4f1a0cbeefc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6055421-f8de-4e98-bd0e-4f1a0cbeefc5-plugins-config-volume\") pod \"b6055421-f8de-4e98-bd0e-4f1a0cbeefc5\" (UID: \"b6055421-f8de-4e98-bd0e-4f1a0cbeefc5\") " Oct 30 12:32:08.164204 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:08.163915 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6055421-f8de-4e98-bd0e-4f1a0cbeefc5-kube-api-access-6d6w6\" (UniqueName: \"kubernetes.io/projected/b6055421-f8de-4e98-bd0e-4f1a0cbeefc5-kube-api-access-6d6w6\") pod \"b6055421-f8de-4e98-bd0e-4f1a0cbeefc5\" (UID: \"b6055421-f8de-4e98-bd0e-4f1a0cbeefc5\") " Oct 30 12:32:08.165309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:08.165276 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6055421-f8de-4e98-bd0e-4f1a0cbeefc5-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:32:08.247111 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:08.238059 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6055421-f8de-4e98-bd0e-4f1a0cbeefc5-kube-api-access-6d6w6" pod "b6055421-f8de-4e98-bd0e-4f1a0cbeefc5" (UID: "b6055421-f8de-4e98-bd0e-4f1a0cbeefc5"). InnerVolumeSpecName "kube-api-access-6d6w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:08.286200 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:08.286158 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d6w6\" (UniqueName: \"kubernetes.io/projected/b6055421-f8de-4e98-bd0e-4f1a0cbeefc5-kube-api-access-6d6w6\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:32:08.529960 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:08.529928 4468 generic.go:356] "Generic (PLEG): container finished" podID="b6055421-f8de-4e98-bd0e-4f1a0cbeefc5" containerID="5d0b9aa6e6b04c55b7144384b93da04556af9f01e31879f7110326a440d80dca" exitCode=0 Oct 30 12:32:08.530272 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:08.530248 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22" event={"ID":"b6055421-f8de-4e98-bd0e-4f1a0cbeefc5","Type":"ContainerDied","Data":"5d0b9aa6e6b04c55b7144384b93da04556af9f01e31879f7110326a440d80dca"} Oct 30 12:32:08.530387 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:08.530343 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22" event={"ID":"b6055421-f8de-4e98-bd0e-4f1a0cbeefc5","Type":"ContainerDied","Data":"ddbd883eca4ba1b25c44bebcd4d3693d380e36b36bf70fcd90b491dddcb844a4"} Oct 30 12:32:08.530387 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:08.530380 4468 scope.go:122] "RemoveContainer" containerID="5d0b9aa6e6b04c55b7144384b93da04556af9f01e31879f7110326a440d80dca" Oct 30 12:32:08.530573 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:08.530557 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22" Oct 30 12:32:08.545517 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:08.545498 4468 scope.go:122] "RemoveContainer" containerID="5d0b9aa6e6b04c55b7144384b93da04556af9f01e31879f7110326a440d80dca" Oct 30 12:32:08.545785 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:32:08.545765 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d0b9aa6e6b04c55b7144384b93da04556af9f01e31879f7110326a440d80dca\": container with ID starting with 5d0b9aa6e6b04c55b7144384b93da04556af9f01e31879f7110326a440d80dca not found: ID does not exist" containerID="5d0b9aa6e6b04c55b7144384b93da04556af9f01e31879f7110326a440d80dca" Oct 30 12:32:08.545845 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:08.545793 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0b9aa6e6b04c55b7144384b93da04556af9f01e31879f7110326a440d80dca"} err="failed to get container status \"5d0b9aa6e6b04c55b7144384b93da04556af9f01e31879f7110326a440d80dca\": rpc error: code = NotFound desc = could not find container \"5d0b9aa6e6b04c55b7144384b93da04556af9f01e31879f7110326a440d80dca\": container with ID starting with 5d0b9aa6e6b04c55b7144384b93da04556af9f01e31879f7110326a440d80dca not found: ID does not exist" Oct 30 12:32:08.592990 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:08.592952 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22"] Oct 30 12:32:08.596855 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:08.596829 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c57af-7cb5949746-79v22"] Oct 30 12:32:09.090666 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:09.090631 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6055421-f8de-4e98-bd0e-4f1a0cbeefc5" path="/var/lib/kubelet/pods/b6055421-f8de-4e98-bd0e-4f1a0cbeefc5/volumes" Oct 30 12:32:19.974966 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.974930 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz"] Oct 30 12:32:19.975505 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975491 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3acebec4-84f8-415a-84f7-f8f7d9391a85" containerName="istio-proxy" Oct 30 12:32:19.975505 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975506 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3acebec4-84f8-415a-84f7-f8f7d9391a85" containerName="istio-proxy" Oct 30 12:32:19.975581 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975516 4468 cpu_manager.go:414] "containerMap: removing container" podUID="b6055421-f8de-4e98-bd0e-4f1a0cbeefc5" containerName="payload-processing" Oct 30 12:32:19.975581 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975522 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6055421-f8de-4e98-bd0e-4f1a0cbeefc5" containerName="payload-processing" Oct 30 12:32:19.975581 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975529 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3acebec4-84f8-415a-84f7-f8f7d9391a85" containerName="istio-proxy" Oct 30 12:32:19.975581 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975535 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3acebec4-84f8-415a-84f7-f8f7d9391a85" containerName="istio-proxy" Oct 30 12:32:19.975581 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975548 4468 cpu_manager.go:414] "containerMap: removing container" podUID="f1d20bcb-72ac-4692-9f8e-a064a2886bce" containerName="maas-api" Oct 30 12:32:19.975581 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975553 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d20bcb-72ac-4692-9f8e-a064a2886bce" containerName="maas-api" Oct 30 12:32:19.975581 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975565 4468 cpu_manager.go:414] "containerMap: removing container" podUID="42e6150b-6f32-4d92-911c-962f73ba0e68" containerName="payload-pre-processing" Oct 30 12:32:19.975581 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975569 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42e6150b-6f32-4d92-911c-962f73ba0e68" containerName="payload-pre-processing" Oct 30 12:32:19.975581 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975578 4468 cpu_manager.go:414] "containerMap: removing container" podUID="17be3a09-0989-4ccb-ae91-4a316e0fdca8" containerName="main" Oct 30 12:32:19.975581 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975583 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17be3a09-0989-4ccb-ae91-4a316e0fdca8" containerName="main" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975594 4468 cpu_manager.go:414] "containerMap: removing container" podUID="0166c51a-8d66-47e5-ba9a-2fb14b342215" containerName="istio-proxy" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975599 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0166c51a-8d66-47e5-ba9a-2fb14b342215" containerName="istio-proxy" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975609 4468 cpu_manager.go:414] "containerMap: removing container" podUID="d11d3c04-4239-41d4-a099-3c6f0d287eaf" containerName="payload-processing" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975614 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d11d3c04-4239-41d4-a099-3c6f0d287eaf" containerName="payload-processing" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975620 4468 cpu_manager.go:414] "containerMap: removing container" podUID="df475a56-3557-40c3-84bb-0e812127ae21" containerName="istio-proxy" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975624 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df475a56-3557-40c3-84bb-0e812127ae21" containerName="istio-proxy" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975632 4468 cpu_manager.go:414] "containerMap: removing container" podUID="0166c51a-8d66-47e5-ba9a-2fb14b342215" containerName="istio-proxy" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975636 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0166c51a-8d66-47e5-ba9a-2fb14b342215" containerName="istio-proxy" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975655 4468 cpu_manager.go:414] "containerMap: removing container" podUID="8a160a20-d339-40f4-9c7a-8457342868d3" containerName="payload-pre-processing" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975659 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a160a20-d339-40f4-9c7a-8457342868d3" containerName="payload-pre-processing" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975735 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="42e6150b-6f32-4d92-911c-962f73ba0e68" containerName="payload-pre-processing" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975742 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="17be3a09-0989-4ccb-ae91-4a316e0fdca8" containerName="main" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975749 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0166c51a-8d66-47e5-ba9a-2fb14b342215" containerName="istio-proxy" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975756 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d11d3c04-4239-41d4-a099-3c6f0d287eaf" containerName="payload-processing" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975763 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="df475a56-3557-40c3-84bb-0e812127ae21" containerName="istio-proxy" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975768 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0166c51a-8d66-47e5-ba9a-2fb14b342215" containerName="istio-proxy" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975777 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="df475a56-3557-40c3-84bb-0e812127ae21" containerName="istio-proxy" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975783 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a160a20-d339-40f4-9c7a-8457342868d3" containerName="payload-pre-processing" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975790 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3acebec4-84f8-415a-84f7-f8f7d9391a85" containerName="istio-proxy" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975794 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6055421-f8de-4e98-bd0e-4f1a0cbeefc5" containerName="payload-processing" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975799 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3acebec4-84f8-415a-84f7-f8f7d9391a85" containerName="istio-proxy" Oct 30 12:32:19.975887 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.975804 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d20bcb-72ac-4692-9f8e-a064a2886bce" containerName="maas-api" Oct 30 12:32:19.982780 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:19.981141 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.011165 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.011127 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc"] Oct 30 12:32:20.012102 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.012058 4468 cpu_manager.go:414] "containerMap: removing container" podUID="df475a56-3557-40c3-84bb-0e812127ae21" containerName="istio-proxy" Oct 30 12:32:20.012227 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.012103 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df475a56-3557-40c3-84bb-0e812127ae21" containerName="istio-proxy" Oct 30 12:32:20.027354 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.027320 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.036520 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.036365 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz"] Oct 30 12:32:20.051160 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.051130 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc"] Oct 30 12:32:20.120212 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.119824 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.120212 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.119885 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.120212 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.120033 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.120212 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.120129 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.120212 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.120173 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkd85\" (UniqueName: \"kubernetes.io/projected/3b5da186-2a29-4bef-87fc-5dbfca607e5e-kube-api-access-mkd85\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.120212 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.120199 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.120674 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.120251 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.120674 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.120310 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.120674 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.120338 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.223093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.223029 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.223424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.223404 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.223541 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.223527 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.223658 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.223645 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.223843 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.223826 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.223999 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.223976 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.224125 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.224112 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5v7r\" (UniqueName: \"kubernetes.io/projected/03176887-19c3-430c-8989-90572b562b75-kube-api-access-b5v7r\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.227426 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.227362 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03176887-19c3-430c-8989-90572b562b75-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.227605 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.227586 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.227745 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.227729 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.227909 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.227891 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03176887-19c3-430c-8989-90572b562b75-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.228040 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.228024 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.228217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.228199 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.228384 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.228368 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.228621 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.228601 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.228778 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.228761 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkd85\" (UniqueName: \"kubernetes.io/projected/3b5da186-2a29-4bef-87fc-5dbfca607e5e-kube-api-access-mkd85\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.228925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.228906 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.229054 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.229039 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03176887-19c3-430c-8989-90572b562b75-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.230127 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.230102 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.231038 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.230890 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.235141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.235120 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.235550 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.235527 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.235992 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.235969 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.250097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.249984 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.250602 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.250581 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.265763 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.265738 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkd85\" (UniqueName: \"kubernetes.io/projected/3b5da186-2a29-4bef-87fc-5dbfca607e5e-kube-api-access-mkd85\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.267583 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.267553 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.328292 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.328256 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:20.330386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.330355 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03176887-19c3-430c-8989-90572b562b75-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.330538 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.330440 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.330538 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.330502 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.330664 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.330561 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.330664 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.330593 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5v7r\" (UniqueName: \"kubernetes.io/projected/03176887-19c3-430c-8989-90572b562b75-kube-api-access-b5v7r\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.330784 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.330674 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03176887-19c3-430c-8989-90572b562b75-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.330784 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.330741 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03176887-19c3-430c-8989-90572b562b75-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.330784 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.330777 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.331010 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.330873 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.331822 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.331383 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.335339 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.333441 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03176887-19c3-430c-8989-90572b562b75-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.340012 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.339394 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.340012 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.339699 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.340012 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.339955 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.350555 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.350496 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.355101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.353389 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03176887-19c3-430c-8989-90572b562b75-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.367315 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.366860 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5v7r\" (UniqueName: \"kubernetes.io/projected/03176887-19c3-430c-8989-90572b562b75-kube-api-access-b5v7r\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.367440 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.367373 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03176887-19c3-430c-8989-90572b562b75-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.393417 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.392919 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:20.756104 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:32:20.750501 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5da186_2a29_4bef_87fc_5dbfca607e5e.slice/crio-4cc017ec76f541a5f7c9de26100a16e8c4669c1390254db87523214948161772 WatchSource:0}: Error finding container 4cc017ec76f541a5f7c9de26100a16e8c4669c1390254db87523214948161772: Status 404 returned error can't find the container with id 4cc017ec76f541a5f7c9de26100a16e8c4669c1390254db87523214948161772 Oct 30 12:32:20.756104 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.751734 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz"] Oct 30 12:32:20.763162 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.763116 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:20.763326 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.763216 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:20.763326 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.763293 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:20.822739 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.822702 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:20.822855 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.822772 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:20.822855 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.822801 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:20.822940 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:20.822903 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc"] Oct 30 12:32:21.469098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.459683 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx"] Oct 30 12:32:21.482435 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.482403 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4"] Oct 30 12:32:21.482701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.482680 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.490215 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.490189 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.575443 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.575404 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.575704 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.575686 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.575811 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.575800 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4zzn\" (UniqueName: \"kubernetes.io/projected/e286532f-d8eb-4c7f-baea-7c71dab94e6e-kube-api-access-v4zzn\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.575976 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.575942 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.576029 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.576000 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.576104 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.576038 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.576156 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.576138 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.576295 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.576270 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.576374 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.576321 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.576430 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.576390 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.576482 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.576462 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.576538 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.576491 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.576538 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.576520 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzw2w\" (UniqueName: \"kubernetes.io/projected/bdc9d563-a787-4dac-b192-9507eaad1b14-kube-api-access-xzw2w\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.576731 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.576711 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.576800 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.576753 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.576862 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.576801 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdc9d563-a787-4dac-b192-9507eaad1b14-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.576862 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.576834 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.576996 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.576973 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.592238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.588471 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx"] Oct 30 12:32:21.601325 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.597361 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4"] Oct 30 12:32:21.678841 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.678808 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.679041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.679030 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.679177 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.679164 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.679257 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.679247 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzw2w\" (UniqueName: \"kubernetes.io/projected/bdc9d563-a787-4dac-b192-9507eaad1b14-kube-api-access-xzw2w\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.679323 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.679315 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.679392 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.679382 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.679465 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.679456 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdc9d563-a787-4dac-b192-9507eaad1b14-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.679530 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.679521 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.679597 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.679590 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.679671 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.679662 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.679746 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.679738 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.679814 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.679806 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4zzn\" (UniqueName: \"kubernetes.io/projected/e286532f-d8eb-4c7f-baea-7c71dab94e6e-kube-api-access-v4zzn\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.679883 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.679875 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.679950 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.679942 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.680021 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.680012 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.680173 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.680158 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.680325 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.680308 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.680453 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.680438 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.684120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.680997 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdc9d563-a787-4dac-b192-9507eaad1b14-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.684120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.681960 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.684120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.682611 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.684120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.683316 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.684120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.683568 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.684120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.683569 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.684120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.683806 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.684120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.684024 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.684120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.684058 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.685845 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.685671 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.689575 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.689540 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" event={"ID":"03176887-19c3-430c-8989-90572b562b75","Type":"ContainerStarted","Data":"0d1efabb70fea1f9a7792010cf958862032d04f738ea640c73dfb68918691d82"} Oct 30 12:32:21.689689 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.689602 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" event={"ID":"03176887-19c3-430c-8989-90572b562b75","Type":"ContainerStarted","Data":"5834d84c9f0ed57ff5a9e9732bfb6b2e08efb37106af70ee25cf629154ea8c49"} Oct 30 12:32:21.692834 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.692805 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" event={"ID":"3b5da186-2a29-4bef-87fc-5dbfca607e5e","Type":"ContainerStarted","Data":"d968c5ce7853b15346671e690ad89043e4a465d02e8aa5c51688ec136dd5991c"} Oct 30 12:32:21.692962 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.692860 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" event={"ID":"3b5da186-2a29-4bef-87fc-5dbfca607e5e","Type":"ContainerStarted","Data":"4cc017ec76f541a5f7c9de26100a16e8c4669c1390254db87523214948161772"} Oct 30 12:32:21.789655 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.789574 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzw2w\" (UniqueName: \"kubernetes.io/projected/bdc9d563-a787-4dac-b192-9507eaad1b14-kube-api-access-xzw2w\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.861965 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.861893 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" podStartSLOduration=2.861870794 podStartE2EDuration="2.861870794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:21.82125692 +0000 UTC m=+2292.855395326" watchObservedRunningTime="2026-10-30 12:32:21.861870794 +0000 UTC m=+2292.896009191" Oct 30 12:32:21.927131 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.920898 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:21.948643 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.946124 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4zzn\" (UniqueName: \"kubernetes.io/projected/e286532f-d8eb-4c7f-baea-7c71dab94e6e-kube-api-access-v4zzn\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.964217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.963462 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:21.978198 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:21.966364 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:22.012098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.007742 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:22.012098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.008564 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:22.046134 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.045989 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" podStartSLOduration=3.045965931 podStartE2EDuration="3.045965931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:21.927681632 +0000 UTC m=+2292.961820030" watchObservedRunningTime="2026-10-30 12:32:22.045965931 +0000 UTC m=+2293.080104328" Oct 30 12:32:22.089676 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.089639 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:22.202135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.202055 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:22.262096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.255402 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:22.337148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.334885 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:22.337148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.335023 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 12:32:22.337148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.335111 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" podUID="3b5da186-2a29-4bef-87fc-5dbfca607e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 12:32:22.393904 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.393870 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:22.394977 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.394954 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 12:32:22.395188 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.395151 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" podUID="03176887-19c3-430c-8989-90572b562b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 12:32:22.772793 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.772474 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx"] Oct 30 12:32:22.774282 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.774241 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:22.774409 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.774330 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:22.774409 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.774372 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:22.921340 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.908249 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4"] Oct 30 12:32:22.941160 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.929226 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:22.941160 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.929326 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:22.941160 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:22.929393 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:23.352147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:23.351752 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 12:32:23.352147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:23.351816 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" podUID="3b5da186-2a29-4bef-87fc-5dbfca607e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 12:32:23.394198 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:23.394163 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 12:32:23.394379 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:23.394352 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" podUID="03176887-19c3-430c-8989-90572b562b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 12:32:23.603445 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:32:23.603323 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:32:23.774096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:23.772493 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" event={"ID":"bdc9d563-a787-4dac-b192-9507eaad1b14","Type":"ContainerStarted","Data":"ef8f4ee77bc4a535e1043d82cab03666d5e5b5c1d0d1f55e7db7ab627aca7bea"} Oct 30 12:32:23.774096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:23.772559 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" event={"ID":"bdc9d563-a787-4dac-b192-9507eaad1b14","Type":"ContainerStarted","Data":"4585c0efe745ed402996841556a4dcb00c2bcab44a589e9ab41edc050c7850c9"} Oct 30 12:32:23.780102 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:23.775500 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" event={"ID":"e286532f-d8eb-4c7f-baea-7c71dab94e6e","Type":"ContainerStarted","Data":"b83423af657a31b3f5749031924bd28e6d43c5006a02fdd6d60c512918a2b549"} Oct 30 12:32:23.780102 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:23.775562 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" event={"ID":"e286532f-d8eb-4c7f-baea-7c71dab94e6e","Type":"ContainerStarted","Data":"43b3279e8d46cb7449157ed63b5e136d6fa7e52a52fe89915013c6ff6cc6aa16"} Oct 30 12:32:23.869111 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:23.868972 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" podStartSLOduration=2.868950876 podStartE2EDuration="2.868950876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:23.825261258 +0000 UTC m=+2294.859399657" watchObservedRunningTime="2026-10-30 12:32:23.868950876 +0000 UTC m=+2294.903089272" Oct 30 12:32:23.883519 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:23.883462 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" podStartSLOduration=2.883448141 podStartE2EDuration="2.883448141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:23.882473906 +0000 UTC m=+2294.916612286" watchObservedRunningTime="2026-10-30 12:32:23.883448141 +0000 UTC m=+2294.917586532" Oct 30 12:32:24.212194 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.212155 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:24.239689 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.239191 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 12:32:24.239689 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.239259 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" podUID="e286532f-d8eb-4c7f-baea-7c71dab94e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 12:32:24.278240 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.278152 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:24.330329 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.306216 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 12:32:24.330329 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.306281 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" podUID="bdc9d563-a787-4dac-b192-9507eaad1b14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 12:32:24.352137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.352095 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 12:32:24.352455 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.352411 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" podUID="3b5da186-2a29-4bef-87fc-5dbfca607e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 12:32:24.405078 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.393628 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 12:32:24.405408 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.405364 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" podUID="03176887-19c3-430c-8989-90572b562b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 12:32:24.576159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.557118 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj"] Oct 30 12:32:24.678260 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.676553 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81958156-cfb9-4398-a8a3-ca72db1036df-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj\" (UID: \"81958156-cfb9-4398-a8a3-ca72db1036df\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj" Oct 30 12:32:24.678260 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.676618 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f67nn\" (UniqueName: \"kubernetes.io/projected/81958156-cfb9-4398-a8a3-ca72db1036df-kube-api-access-f67nn\") pod \"payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj\" (UID: \"81958156-cfb9-4398-a8a3-ca72db1036df\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj" Oct 30 12:32:24.679119 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.679100 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj" Oct 30 12:32:24.708460 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.702732 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-a124ab-dockercfg-rlwj9\"" Oct 30 12:32:24.708460 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.702818 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-a124ab\"" Oct 30 12:32:24.763148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.757605 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj"] Oct 30 12:32:24.792234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.792184 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81958156-cfb9-4398-a8a3-ca72db1036df-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj\" (UID: \"81958156-cfb9-4398-a8a3-ca72db1036df\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj" Oct 30 12:32:24.792859 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.792833 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f67nn\" (UniqueName: \"kubernetes.io/projected/81958156-cfb9-4398-a8a3-ca72db1036df-kube-api-access-f67nn\") pod \"payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj\" (UID: \"81958156-cfb9-4398-a8a3-ca72db1036df\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj" Oct 30 12:32:24.817092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.794505 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81958156-cfb9-4398-a8a3-ca72db1036df-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj\" (UID: \"81958156-cfb9-4398-a8a3-ca72db1036df\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj" Oct 30 12:32:24.867283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:24.867171 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f67nn\" (UniqueName: \"kubernetes.io/projected/81958156-cfb9-4398-a8a3-ca72db1036df-kube-api-access-f67nn\") pod \"payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj\" (UID: \"81958156-cfb9-4398-a8a3-ca72db1036df\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj" Oct 30 12:32:25.251095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:25.250198 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 12:32:25.251095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:25.250259 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" podUID="e286532f-d8eb-4c7f-baea-7c71dab94e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 12:32:25.265095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:25.251870 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj" Oct 30 12:32:25.317101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:25.281498 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 12:32:25.317101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:25.281561 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" podUID="bdc9d563-a787-4dac-b192-9507eaad1b14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 12:32:25.448003 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:25.447962 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 12:32:25.448363 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:25.448334 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" podUID="3b5da186-2a29-4bef-87fc-5dbfca607e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 12:32:25.483100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:25.473157 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:25.513148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:25.511221 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 12:32:25.513148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:25.511295 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" podUID="03176887-19c3-430c-8989-90572b562b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 12:32:25.585114 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:25.581467 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 12:32:25.690921 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:25.690872 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:25.861393 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:25.861283 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:25.920762 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:25.920714 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:26.294750 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.294710 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 12:32:26.310201 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.310163 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" podUID="e286532f-d8eb-4c7f-baea-7c71dab94e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 12:32:26.322277 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.295495 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded" Oct 30 12:32:26.322793 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.295032 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 12:32:26.322986 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.322957 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" podUID="bdc9d563-a787-4dac-b192-9507eaad1b14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 12:32:26.359905 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.359869 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 12:32:26.365008 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.364964 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" podUID="3b5da186-2a29-4bef-87fc-5dbfca607e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 12:32:26.519770 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.519724 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:26.520401 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.520369 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 12:32:26.521907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.521883 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 12:32:26.522105 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.522082 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" podUID="03176887-19c3-430c-8989-90572b562b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 12:32:26.573008 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.572900 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj"] Oct 30 12:32:26.689106 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.683172 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:26.689106 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.683655 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:32:26.735094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.709142 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:26.735094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.709574 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:32:26.767926 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.765925 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:26.928244 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.875853 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:26.982678 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.927714 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:26.982678 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.968710 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:32:26.982678 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.968774 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:32:26.982678 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:26.971621 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:27.250156 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.235991 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 12:32:27.250156 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.236052 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" podUID="e286532f-d8eb-4c7f-baea-7c71dab94e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 12:32:27.325730 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.282185 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 12:32:27.325730 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.282242 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" podUID="bdc9d563-a787-4dac-b192-9507eaad1b14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 12:32:27.496098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.367720 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 12:32:27.496098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.367772 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" podUID="3b5da186-2a29-4bef-87fc-5dbfca607e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 12:32:27.496098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.441477 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 12:32:27.496098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.441533 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" podUID="03176887-19c3-430c-8989-90572b562b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 12:32:27.500505 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.500397 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:27.519386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.519344 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj" event={"ID":"81958156-cfb9-4398-a8a3-ca72db1036df","Type":"ContainerStarted","Data":"68dc115ef9dc6f3cd21a07c346d5711aec25d238e8329d5872067a75513898b0"} Oct 30 12:32:27.522385 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.521262 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:27.524377 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.521297 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 30 12:32:27.531102 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.527058 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:32:27.569095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.563838 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:32:27.683571 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.683507 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:27.721550 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.721497 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:27.773161 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.769292 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 12:32:27.773534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.773496 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:32:27.886758 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.886728 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:32:27.981104 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.977403 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:28.000105 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:27.987431 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 12:32:28.202215 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:28.198789 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:32:28.287669 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:28.284860 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 12:32:28.287669 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:28.284925 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" podUID="e286532f-d8eb-4c7f-baea-7c71dab94e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 12:32:28.310941 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:28.310910 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 12:32:28.315231 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:28.315149 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" podUID="bdc9d563-a787-4dac-b192-9507eaad1b14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 12:32:28.315530 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:28.315512 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:32:28.315684 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:28.315674 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:32:28.373103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:28.363149 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 12:32:28.373103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:28.363204 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" podUID="3b5da186-2a29-4bef-87fc-5dbfca607e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 12:32:28.392759 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:28.392726 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:32:28.395346 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:28.394962 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 12:32:28.395346 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:28.395016 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" podUID="03176887-19c3-430c-8989-90572b562b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 12:32:28.395546 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:28.395483 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:32:28.401368 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:28.401267 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:32:29.217101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:29.216502 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 12:32:29.217101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:29.216569 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" podUID="e286532f-d8eb-4c7f-baea-7c71dab94e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 12:32:29.274096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:29.273493 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 12:32:29.274096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:29.273560 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" podUID="bdc9d563-a787-4dac-b192-9507eaad1b14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 12:32:29.330092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:29.329737 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 12:32:29.330092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:29.329801 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" podUID="3b5da186-2a29-4bef-87fc-5dbfca607e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 12:32:29.394293 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:29.394182 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 12:32:29.394293 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:29.394249 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" podUID="03176887-19c3-430c-8989-90572b562b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 12:32:29.645666 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:29.645574 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj" event={"ID":"81958156-cfb9-4398-a8a3-ca72db1036df","Type":"ContainerStarted","Data":"e44b720363568dbf011c0226383474f63489c8d0b4452b198d04a578dae4d6c9"} Oct 30 12:32:29.676384 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:29.676352 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj" Oct 30 12:32:29.831100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:29.818507 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj" podStartSLOduration=5.818486845 podStartE2EDuration="5.818486845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:29.818462532 +0000 UTC m=+2300.852600930" watchObservedRunningTime="2026-10-30 12:32:29.818486845 +0000 UTC m=+2300.852625242" Oct 30 12:32:30.207960 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:30.207531 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 12:32:30.207960 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:30.207593 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" podUID="e286532f-d8eb-4c7f-baea-7c71dab94e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 12:32:30.320204 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:30.277820 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 12:32:30.378093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:30.373162 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" podUID="bdc9d563-a787-4dac-b192-9507eaad1b14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 12:32:30.378093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:30.348500 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl"] Oct 30 12:32:30.393273 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:30.393241 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl"] Oct 30 12:32:30.393606 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:30.393590 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl" Oct 30 12:32:30.476931 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:30.474651 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-2461e8-dockercfg-xkclw\"" Oct 30 12:32:30.476931 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:30.474748 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-2461e8\"" Oct 30 12:32:30.494949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:30.494912 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns2j7\" (UniqueName: \"kubernetes.io/projected/efb3943e-edd9-4e48-affc-17631ef96950-kube-api-access-ns2j7\") pod \"payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl\" (UID: \"efb3943e-edd9-4e48-affc-17631ef96950\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl" Oct 30 12:32:30.495355 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:30.495333 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efb3943e-edd9-4e48-affc-17631ef96950-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl\" (UID: \"efb3943e-edd9-4e48-affc-17631ef96950\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl" Oct 30 12:32:30.597135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:30.597087 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efb3943e-edd9-4e48-affc-17631ef96950-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl\" (UID: \"efb3943e-edd9-4e48-affc-17631ef96950\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl" Oct 30 12:32:30.623392 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:30.623348 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns2j7\" (UniqueName: \"kubernetes.io/projected/efb3943e-edd9-4e48-affc-17631ef96950-kube-api-access-ns2j7\") pod \"payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl\" (UID: \"efb3943e-edd9-4e48-affc-17631ef96950\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl" Oct 30 12:32:30.646094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:30.624963 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efb3943e-edd9-4e48-affc-17631ef96950-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl\" (UID: \"efb3943e-edd9-4e48-affc-17631ef96950\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl" Oct 30 12:32:30.712093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:30.708097 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns2j7\" (UniqueName: \"kubernetes.io/projected/efb3943e-edd9-4e48-affc-17631ef96950-kube-api-access-ns2j7\") pod \"payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl\" (UID: \"efb3943e-edd9-4e48-affc-17631ef96950\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl" Oct 30 12:32:30.878025 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:30.877775 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl" Oct 30 12:32:30.960903 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:30.959579 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:30.961159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:30.961121 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:30.962129 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:30.962027 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:32:31.084302 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:31.084263 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:31.085945 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:31.085916 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:31.112144 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:31.109987 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:32:31.206121 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:31.203347 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 12:32:31.206121 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:31.203409 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" podUID="e286532f-d8eb-4c7f-baea-7c71dab94e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 12:32:31.268013 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:31.267976 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 12:32:31.268216 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:31.268047 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" podUID="bdc9d563-a787-4dac-b192-9507eaad1b14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 12:32:31.285010 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:31.284977 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl"] Oct 30 12:32:31.286563 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:32:31.286527 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb3943e_edd9_4e48_affc_17631ef96950.slice/crio-ee7d46af8149fcd1459a7ea538d6e3115f91a4d8b4f144e8de5e28f6d6ff4247 WatchSource:0}: Error finding container ee7d46af8149fcd1459a7ea538d6e3115f91a4d8b4f144e8de5e28f6d6ff4247: Status 404 returned error can't find the container with id ee7d46af8149fcd1459a7ea538d6e3115f91a4d8b4f144e8de5e28f6d6ff4247 Oct 30 12:32:31.695976 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:31.695926 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl" event={"ID":"efb3943e-edd9-4e48-affc-17631ef96950","Type":"ContainerStarted","Data":"8739320b8913d37dd5d073d29b82a49a9c10d8d4189cee92a7f6686a240cdb1d"} Oct 30 12:32:31.696429 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:31.696035 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl" event={"ID":"efb3943e-edd9-4e48-affc-17631ef96950","Type":"ContainerStarted","Data":"ee7d46af8149fcd1459a7ea538d6e3115f91a4d8b4f144e8de5e28f6d6ff4247"} Oct 30 12:32:31.742762 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:31.742643 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl" podStartSLOduration=2.742625084 podStartE2EDuration="2.742625084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:31.740277531 +0000 UTC m=+2302.774415928" watchObservedRunningTime="2026-10-30 12:32:31.742625084 +0000 UTC m=+2302.776763482" Oct 30 12:32:32.631301 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:32.631266 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:32.631764 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:32.631740 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:32.632637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:32.632613 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:32:32.708222 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:32.708191 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl" Oct 30 12:32:32.769091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:32.767645 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:32.771108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:32.769381 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:32.771108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:32.770432 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:32:36.961885 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:36.961782 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:36.983610 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:36.983508 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:41.699748 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:41.699711 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj" Oct 30 12:32:43.719876 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:43.719848 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl" Oct 30 12:32:43.900231 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:43.900192 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7"] Oct 30 12:32:43.900429 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:43.900403 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" podUID="634f6508-78a2-46d9-b94d-3d9a4b184441" containerName="istio-proxy" containerID="cri-o://4bde4b1503dc339fd13fe30cd398b428c8116514c8165b7b1297cc17fb1fbeb7" gracePeriod=30 Oct 30 12:32:47.563810 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:47.563772 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:32:47.564315 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:47.563855 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" podUID="634f6508-78a2-46d9-b94d-3d9a4b184441" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:32:49.172475 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.172407 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:32:49.242695 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.242650 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-token\" (UniqueName: \"kubernetes.io/projected/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-token\") pod \"634f6508-78a2-46d9-b94d-3d9a4b184441\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " Oct 30 12:32:49.242890 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.242710 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-credential-socket\") pod \"634f6508-78a2-46d9-b94d-3d9a4b184441\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " Oct 30 12:32:49.242890 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.242743 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-podinfo\") pod \"634f6508-78a2-46d9-b94d-3d9a4b184441\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " Oct 30 12:32:49.242890 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.242826 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-workload-socket\") pod \"634f6508-78a2-46d9-b94d-3d9a4b184441\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " Oct 30 12:32:49.243088 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.242941 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-envoy\") pod \"634f6508-78a2-46d9-b94d-3d9a4b184441\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " Oct 30 12:32:49.243088 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.242975 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/634f6508-78a2-46d9-b94d-3d9a4b184441-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/634f6508-78a2-46d9-b94d-3d9a4b184441-istiod-ca-cert\") pod \"634f6508-78a2-46d9-b94d-3d9a4b184441\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " Oct 30 12:32:49.243203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.243153 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/634f6508-78a2-46d9-b94d-3d9a4b184441-kube-api-access-tk9k8\" (UniqueName: \"kubernetes.io/projected/634f6508-78a2-46d9-b94d-3d9a4b184441-kube-api-access-tk9k8\") pod \"634f6508-78a2-46d9-b94d-3d9a4b184441\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " Oct 30 12:32:49.243203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.243188 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-workload-certs\") pod \"634f6508-78a2-46d9-b94d-3d9a4b184441\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " Oct 30 12:32:49.243293 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.243215 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-data\") pod \"634f6508-78a2-46d9-b94d-3d9a4b184441\" (UID: \"634f6508-78a2-46d9-b94d-3d9a4b184441\") " Oct 30 12:32:49.244928 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.244891 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-data" pod "634f6508-78a2-46d9-b94d-3d9a4b184441" (UID: "634f6508-78a2-46d9-b94d-3d9a4b184441"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:49.245534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.245507 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634f6508-78a2-46d9-b94d-3d9a4b184441-istiod-ca-cert" pod "634f6508-78a2-46d9-b94d-3d9a4b184441" (UID: "634f6508-78a2-46d9-b94d-3d9a4b184441"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:49.245791 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.245767 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-credential-socket" pod "634f6508-78a2-46d9-b94d-3d9a4b184441" (UID: "634f6508-78a2-46d9-b94d-3d9a4b184441"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:49.246844 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.246809 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-workload-certs" pod "634f6508-78a2-46d9-b94d-3d9a4b184441" (UID: "634f6508-78a2-46d9-b94d-3d9a4b184441"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:49.247317 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.247287 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-token" pod "634f6508-78a2-46d9-b94d-3d9a4b184441" (UID: "634f6508-78a2-46d9-b94d-3d9a4b184441"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:49.247732 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.247709 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-workload-socket" pod "634f6508-78a2-46d9-b94d-3d9a4b184441" (UID: "634f6508-78a2-46d9-b94d-3d9a4b184441"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:49.248808 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.248785 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-podinfo" pod "634f6508-78a2-46d9-b94d-3d9a4b184441" (UID: "634f6508-78a2-46d9-b94d-3d9a4b184441"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:32:49.250196 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.250173 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-envoy" pod "634f6508-78a2-46d9-b94d-3d9a4b184441" (UID: "634f6508-78a2-46d9-b94d-3d9a4b184441"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:49.252020 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.251995 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634f6508-78a2-46d9-b94d-3d9a4b184441-kube-api-access-tk9k8" pod "634f6508-78a2-46d9-b94d-3d9a4b184441" (UID: "634f6508-78a2-46d9-b94d-3d9a4b184441"). InnerVolumeSpecName "kube-api-access-tk9k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:49.349288 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.349257 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:32:49.349288 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.349286 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:32:49.349288 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.349296 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:32:49.349534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.349304 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:32:49.349534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.349315 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/634f6508-78a2-46d9-b94d-3d9a4b184441-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:32:49.349534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.349323 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk9k8\" (UniqueName: \"kubernetes.io/projected/634f6508-78a2-46d9-b94d-3d9a4b184441-kube-api-access-tk9k8\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:32:49.349534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.349332 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:32:49.349534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.349341 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:32:49.349534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.349350 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/634f6508-78a2-46d9-b94d-3d9a4b184441-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:32:49.629254 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.629162 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr"] Oct 30 12:32:49.898484 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.898392 4468 generic.go:356] "Generic (PLEG): container finished" podID="634f6508-78a2-46d9-b94d-3d9a4b184441" containerID="4bde4b1503dc339fd13fe30cd398b428c8116514c8165b7b1297cc17fb1fbeb7" exitCode=0 Oct 30 12:32:49.898484 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.898445 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" Oct 30 12:32:49.898484 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.898460 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" event={"ID":"634f6508-78a2-46d9-b94d-3d9a4b184441","Type":"ContainerDied","Data":"4bde4b1503dc339fd13fe30cd398b428c8116514c8165b7b1297cc17fb1fbeb7"} Oct 30 12:32:49.898744 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.898513 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7" event={"ID":"634f6508-78a2-46d9-b94d-3d9a4b184441","Type":"ContainerDied","Data":"f03049407cde8220cdc70cfa7b1d61fb9ea9ac1e3d9cf314e753f5d99138837d"} Oct 30 12:32:49.898744 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.898543 4468 scope.go:122] "RemoveContainer" containerID="4bde4b1503dc339fd13fe30cd398b428c8116514c8165b7b1297cc17fb1fbeb7" Oct 30 12:32:49.913237 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.913213 4468 scope.go:122] "RemoveContainer" containerID="4bde4b1503dc339fd13fe30cd398b428c8116514c8165b7b1297cc17fb1fbeb7" Oct 30 12:32:49.913773 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:32:49.913742 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bde4b1503dc339fd13fe30cd398b428c8116514c8165b7b1297cc17fb1fbeb7\": container with ID starting with 4bde4b1503dc339fd13fe30cd398b428c8116514c8165b7b1297cc17fb1fbeb7 not found: ID does not exist" containerID="4bde4b1503dc339fd13fe30cd398b428c8116514c8165b7b1297cc17fb1fbeb7" Oct 30 12:32:49.913859 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.913781 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bde4b1503dc339fd13fe30cd398b428c8116514c8165b7b1297cc17fb1fbeb7"} err="failed to get container status \"4bde4b1503dc339fd13fe30cd398b428c8116514c8165b7b1297cc17fb1fbeb7\": rpc error: code = NotFound desc = could not find container \"4bde4b1503dc339fd13fe30cd398b428c8116514c8165b7b1297cc17fb1fbeb7\": container with ID starting with 4bde4b1503dc339fd13fe30cd398b428c8116514c8165b7b1297cc17fb1fbeb7 not found: ID does not exist" Oct 30 12:32:49.956953 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.956921 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7"] Oct 30 12:32:49.962719 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:49.962675 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kp7x7"] Oct 30 12:32:50.084910 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.084876 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2"] Oct 30 12:32:50.085548 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.085531 4468 cpu_manager.go:414] "containerMap: removing container" podUID="634f6508-78a2-46d9-b94d-3d9a4b184441" containerName="istio-proxy" Oct 30 12:32:50.085620 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.085549 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="634f6508-78a2-46d9-b94d-3d9a4b184441" containerName="istio-proxy" Oct 30 12:32:50.085694 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.085682 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="634f6508-78a2-46d9-b94d-3d9a4b184441" containerName="istio-proxy" Oct 30 12:32:50.090809 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.090781 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.113122 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.113087 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2"] Oct 30 12:32:50.265011 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.263028 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.265011 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.263115 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f34a4f2-182f-48c8-9be1-442a72d04c71-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.265011 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.263174 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.265011 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.263395 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-829pw\" (UniqueName: \"kubernetes.io/projected/0f34a4f2-182f-48c8-9be1-442a72d04c71-kube-api-access-829pw\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.265011 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.263513 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.265011 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.263566 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.265011 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.263614 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.265011 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.263787 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.265011 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.263822 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.364902 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.364857 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f34a4f2-182f-48c8-9be1-442a72d04c71-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.365138 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.364910 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.365138 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.364971 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-829pw\" (UniqueName: \"kubernetes.io/projected/0f34a4f2-182f-48c8-9be1-442a72d04c71-kube-api-access-829pw\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.365138 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.365008 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.365138 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.365043 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.365138 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.365086 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.365430 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.365184 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.365430 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.365302 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.365430 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.365392 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.365586 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.365485 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.365586 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.365509 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.365764 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.365746 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.365910 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.365886 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.366250 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.366230 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f34a4f2-182f-48c8-9be1-442a72d04c71-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.367735 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.367707 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.367833 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.367734 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.389729 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.389701 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.389845 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.389783 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-829pw\" (UniqueName: \"kubernetes.io/projected/0f34a4f2-182f-48c8-9be1-442a72d04c71-kube-api-access-829pw\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.413683 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.413653 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:50.738260 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:32:50.738223 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f34a4f2_182f_48c8_9be1_442a72d04c71.slice/crio-07dbb6ccbee1edcc3a90cf2b498ca349572e8c1a080b0423e96028519e56de4b WatchSource:0}: Error finding container 07dbb6ccbee1edcc3a90cf2b498ca349572e8c1a080b0423e96028519e56de4b: Status 404 returned error can't find the container with id 07dbb6ccbee1edcc3a90cf2b498ca349572e8c1a080b0423e96028519e56de4b Oct 30 12:32:50.738871 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.738846 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2"] Oct 30 12:32:50.740756 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.740722 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:50.740875 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.740787 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:50.740875 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.740815 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:50.913376 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.913329 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" event={"ID":"0f34a4f2-182f-48c8-9be1-442a72d04c71","Type":"ContainerStarted","Data":"23d17e3f7da9bf80562519af20ea7657d62907435c7ef179be837c1ab780918f"} Oct 30 12:32:50.913543 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.913428 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" event={"ID":"0f34a4f2-182f-48c8-9be1-442a72d04c71","Type":"ContainerStarted","Data":"07dbb6ccbee1edcc3a90cf2b498ca349572e8c1a080b0423e96028519e56de4b"} Oct 30 12:32:50.962854 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:50.962806 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" podStartSLOduration=0.96279227 podStartE2EDuration="962.79227ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:50.962616066 +0000 UTC m=+2321.996754461" watchObservedRunningTime="2026-10-30 12:32:50.96279227 +0000 UTC m=+2321.996930663" Oct 30 12:32:51.091421 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.091332 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="634f6508-78a2-46d9-b94d-3d9a4b184441" path="/var/lib/kubelet/pods/634f6508-78a2-46d9-b94d-3d9a4b184441/volumes" Oct 30 12:32:51.414236 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.414201 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:51.416033 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.416006 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 12:32:51.416192 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.416053 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" podUID="0f34a4f2-182f-48c8-9be1-442a72d04c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 12:32:51.690159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.688720 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm"] Oct 30 12:32:51.695092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.694438 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.755515 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.755468 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm"] Oct 30 12:32:51.784983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.784943 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.785273 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.785254 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.785416 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.785398 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e88604-3549-4999-aefa-9bccc1042c87-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.785604 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.785585 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.785778 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.785758 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e88604-3549-4999-aefa-9bccc1042c87-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.785912 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.785896 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.786035 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.786022 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9gzl\" (UniqueName: \"kubernetes.io/projected/87e88604-3549-4999-aefa-9bccc1042c87-kube-api-access-b9gzl\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.786268 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.786207 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.786414 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.786399 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e88604-3549-4999-aefa-9bccc1042c87-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.888422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.888389 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.888614 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.888436 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e88604-3549-4999-aefa-9bccc1042c87-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.888614 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.888515 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.888614 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.888538 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.888614 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.888569 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e88604-3549-4999-aefa-9bccc1042c87-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.888790 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.888644 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.888790 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.888713 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e88604-3549-4999-aefa-9bccc1042c87-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.888790 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.888752 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.888940 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.888789 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9gzl\" (UniqueName: \"kubernetes.io/projected/87e88604-3549-4999-aefa-9bccc1042c87-kube-api-access-b9gzl\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.889041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.889013 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.889135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.889109 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.889358 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.889327 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.889479 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.889382 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.889637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.889613 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e88604-3549-4999-aefa-9bccc1042c87-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.891337 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.891309 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.891423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.891369 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e88604-3549-4999-aefa-9bccc1042c87-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.933480 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.933440 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e88604-3549-4999-aefa-9bccc1042c87-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:51.934397 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:51.934368 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9gzl\" (UniqueName: \"kubernetes.io/projected/87e88604-3549-4999-aefa-9bccc1042c87-kube-api-access-b9gzl\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:52.087482 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:52.087392 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:52.428678 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:52.428570 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 12:32:52.428678 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:52.428625 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" podUID="0f34a4f2-182f-48c8-9be1-442a72d04c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 12:32:52.535470 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:52.535351 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm"] Oct 30 12:32:52.538647 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:52.538606 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:52.538757 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:52.538714 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:52.538819 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:52.538782 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:52.947111 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:52.947046 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" event={"ID":"87e88604-3549-4999-aefa-9bccc1042c87","Type":"ContainerStarted","Data":"c5ed3d61a9e7025b96bc43fde405c331a545680968771a4ec5ade6895b292df3"} Oct 30 12:32:52.947294 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:52.947139 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" event={"ID":"87e88604-3549-4999-aefa-9bccc1042c87","Type":"ContainerStarted","Data":"1332fe13ae79948b4d9c9f9b6c9e0d59049a550fac10f2d1db8a09453b0bb19a"} Oct 30 12:32:52.997445 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:52.997388 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" podStartSLOduration=1.997374183 podStartE2EDuration="1.997374183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:52.996980767 +0000 UTC m=+2324.031119164" watchObservedRunningTime="2026-10-30 12:32:52.997374183 +0000 UTC m=+2324.031512578" Oct 30 12:32:53.089367 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:53.089335 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 12:32:53.089567 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:53.089394 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" podUID="87e88604-3549-4999-aefa-9bccc1042c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 12:32:53.090787 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:53.090761 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:32:53.428096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:53.424268 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 12:32:53.428096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:53.424328 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" podUID="0f34a4f2-182f-48c8-9be1-442a72d04c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 12:32:53.620090 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:32:53.609341 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:32:54.093094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:54.088559 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 12:32:54.093094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:54.088624 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" podUID="87e88604-3549-4999-aefa-9bccc1042c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 12:32:54.417093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:54.416434 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 12:32:54.417093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:54.416510 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" podUID="0f34a4f2-182f-48c8-9be1-442a72d04c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 12:32:54.631091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:54.630145 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" podUID="b36f2ed4-3517-4110-ab7a-8cfae8243bab" containerName="payload-pre-processing" containerID="cri-o://75a7c0baa2537a337a615134ca9d593d22b8c3ab8f9d6d4acf6cad38922e1a5c" gracePeriod=25 Oct 30 12:32:55.087898 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:55.087858 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 12:32:55.088138 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:55.087939 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" podUID="87e88604-3549-4999-aefa-9bccc1042c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 12:32:55.426289 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:55.426251 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 12:32:55.426593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:55.426563 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" podUID="0f34a4f2-182f-48c8-9be1-442a72d04c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 12:32:56.088499 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:56.088460 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 12:32:56.088896 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:56.088516 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" podUID="87e88604-3549-4999-aefa-9bccc1042c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 12:32:56.429093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:56.422437 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 12:32:56.429093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:56.422503 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" podUID="0f34a4f2-182f-48c8-9be1-442a72d04c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 12:32:57.087997 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:57.087962 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 12:32:57.088198 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:57.088030 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" podUID="87e88604-3549-4999-aefa-9bccc1042c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 12:32:57.419094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:57.414421 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 12:32:57.419094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:57.414483 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" podUID="0f34a4f2-182f-48c8-9be1-442a72d04c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 12:32:58.088564 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:58.088526 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 12:32:58.088757 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:58.088590 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" podUID="87e88604-3549-4999-aefa-9bccc1042c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 12:32:58.640482 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:58.639353 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:58.640482 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:58.640397 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:58.641762 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:58.641743 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:32:59.087870 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:59.087841 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 12:32:59.088110 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:32:59.087895 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" podUID="87e88604-3549-4999-aefa-9bccc1042c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 12:33:00.088329 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:00.088288 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 12:33:00.088881 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:00.088370 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" podUID="87e88604-3549-4999-aefa-9bccc1042c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 12:33:01.295094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:01.294143 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" podUID="b36f2ed4-3517-4110-ab7a-8cfae8243bab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:33:01.535093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:01.519348 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:33:01.535093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:01.520784 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:33:01.556154 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:01.540715 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:33:04.327386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.327349 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x"] Oct 30 12:33:04.332929 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.332897 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.361433 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.361397 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x"] Oct 30 12:33:04.364111 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.364083 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-cc08f1-openshift-default-dockercfg-rrh7d\"" Oct 30 12:33:04.379357 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.379321 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-envoy\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.379357 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.379360 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s767l\" (UniqueName: \"kubernetes.io/projected/f9f83c6f-04b8-46a1-824a-da232fe7c23d-kube-api-access-s767l\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.379575 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.379426 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-data\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.379575 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.379502 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-token\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.379575 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.379551 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istiod-ca-cert\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.379736 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.379595 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-podinfo\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.379736 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.379627 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-workload-socket\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.379736 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.379664 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-workload-certs\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.379863 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.379796 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-credential-socket\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.480816 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.480780 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-credential-socket\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.481010 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.480839 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-envoy\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.481010 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.480873 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s767l\" (UniqueName: \"kubernetes.io/projected/f9f83c6f-04b8-46a1-824a-da232fe7c23d-kube-api-access-s767l\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.481010 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.480915 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-data\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.481220 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.481014 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-token\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.481220 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.481060 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istiod-ca-cert\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.481220 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.481137 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-podinfo\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.481220 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.481169 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-workload-socket\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.481220 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.481205 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-workload-certs\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.481675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.481643 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-workload-socket\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.481780 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.481717 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-data\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.481780 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.481755 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-credential-socket\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.481902 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.481758 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-workload-certs\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.481961 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.481929 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istiod-ca-cert\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.483747 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.483727 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-envoy\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.484015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.483994 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-podinfo\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.507281 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.506529 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s767l\" (UniqueName: \"kubernetes.io/projected/f9f83c6f-04b8-46a1-824a-da232fe7c23d-kube-api-access-s767l\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.507476 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.507353 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-token\") pod \"e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:04.657634 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:04.657268 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:05.023366 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:33:05.023334 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f83c6f_04b8_46a1_824a_da232fe7c23d.slice/crio-26d03b3e633f13de27dc380823444aaef8a5b0e5107d205020c138efb557f6ad WatchSource:0}: Error finding container 26d03b3e633f13de27dc380823444aaef8a5b0e5107d205020c138efb557f6ad: Status 404 returned error can't find the container with id 26d03b3e633f13de27dc380823444aaef8a5b0e5107d205020c138efb557f6ad Oct 30 12:33:05.025854 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:05.025828 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x"] Oct 30 12:33:05.026386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:05.026346 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:33:05.026454 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:05.026437 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:33:05.026497 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:05.026482 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:33:05.222474 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:05.222441 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" event={"ID":"f9f83c6f-04b8-46a1-824a-da232fe7c23d","Type":"ContainerStarted","Data":"c0fa5408a45f5b4d0b5d12f2afb6f025f883d5dff83297a912598d0734f468cf"} Oct 30 12:33:05.222665 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:05.222497 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" event={"ID":"f9f83c6f-04b8-46a1-824a-da232fe7c23d","Type":"ContainerStarted","Data":"26d03b3e633f13de27dc380823444aaef8a5b0e5107d205020c138efb557f6ad"} Oct 30 12:33:05.273199 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:05.273099 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" podStartSLOduration=1.273084639 podStartE2EDuration="1.273084639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:33:05.270859722 +0000 UTC m=+2336.304998118" watchObservedRunningTime="2026-10-30 12:33:05.273084639 +0000 UTC m=+2336.307223095" Oct 30 12:33:05.658477 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:05.658439 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:06.664470 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:06.664438 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:07.235236 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:07.235203 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:07.236463 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:07.236441 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:09.494906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.494872 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns"] Oct 30 12:33:09.499464 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.499438 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns" Oct 30 12:33:09.518636 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.518605 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns"] Oct 30 12:33:09.525709 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.525682 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-cc08f1\"" Oct 30 12:33:09.525879 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.525786 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-cc08f1-dockercfg-trktc\"" Oct 30 12:33:09.549907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.549873 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp"] Oct 30 12:33:09.555212 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.555181 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp" Oct 30 12:33:09.579952 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.579917 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp"] Oct 30 12:33:09.583819 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.583792 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84vrp\" (UniqueName: \"kubernetes.io/projected/4b5e5158-cd9b-4722-bd22-b0ab3f3e299d-kube-api-access-84vrp\") pod \"payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns\" (UID: \"4b5e5158-cd9b-4722-bd22-b0ab3f3e299d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns" Oct 30 12:33:09.583962 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.583925 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b5e5158-cd9b-4722-bd22-b0ab3f3e299d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns\" (UID: \"4b5e5158-cd9b-4722-bd22-b0ab3f3e299d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns" Oct 30 12:33:09.738092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.720247 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r7fj\" (UniqueName: \"kubernetes.io/projected/8cd2c069-88b7-4611-b517-c836547c29f2-kube-api-access-7r7fj\") pod \"payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp\" (UID: \"8cd2c069-88b7-4611-b517-c836547c29f2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp" Oct 30 12:33:09.819330 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.819240 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd2c069-88b7-4611-b517-c836547c29f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp\" (UID: \"8cd2c069-88b7-4611-b517-c836547c29f2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp" Oct 30 12:33:09.821420 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.821381 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84vrp\" (UniqueName: \"kubernetes.io/projected/4b5e5158-cd9b-4722-bd22-b0ab3f3e299d-kube-api-access-84vrp\") pod \"payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns\" (UID: \"4b5e5158-cd9b-4722-bd22-b0ab3f3e299d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns" Oct 30 12:33:09.822007 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.821971 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b5e5158-cd9b-4722-bd22-b0ab3f3e299d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns\" (UID: \"4b5e5158-cd9b-4722-bd22-b0ab3f3e299d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns" Oct 30 12:33:09.822857 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.822833 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b5e5158-cd9b-4722-bd22-b0ab3f3e299d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns\" (UID: \"4b5e5158-cd9b-4722-bd22-b0ab3f3e299d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns" Oct 30 12:33:09.846714 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.846683 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84vrp\" (UniqueName: \"kubernetes.io/projected/4b5e5158-cd9b-4722-bd22-b0ab3f3e299d-kube-api-access-84vrp\") pod \"payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns\" (UID: \"4b5e5158-cd9b-4722-bd22-b0ab3f3e299d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns" Oct 30 12:33:09.924444 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.924403 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r7fj\" (UniqueName: \"kubernetes.io/projected/8cd2c069-88b7-4611-b517-c836547c29f2-kube-api-access-7r7fj\") pod \"payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp\" (UID: \"8cd2c069-88b7-4611-b517-c836547c29f2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp" Oct 30 12:33:09.924444 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.924448 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd2c069-88b7-4611-b517-c836547c29f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp\" (UID: \"8cd2c069-88b7-4611-b517-c836547c29f2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp" Oct 30 12:33:09.925299 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.925271 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd2c069-88b7-4611-b517-c836547c29f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp\" (UID: \"8cd2c069-88b7-4611-b517-c836547c29f2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp" Oct 30 12:33:09.949733 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:09.949695 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r7fj\" (UniqueName: \"kubernetes.io/projected/8cd2c069-88b7-4611-b517-c836547c29f2-kube-api-access-7r7fj\") pod \"payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp\" (UID: \"8cd2c069-88b7-4611-b517-c836547c29f2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp" Oct 30 12:33:10.118008 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:10.117926 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns" Oct 30 12:33:10.176554 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:10.176521 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp" Oct 30 12:33:10.420818 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:33:10.420784 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b5e5158_cd9b_4722_bd22_b0ab3f3e299d.slice/crio-159ee1ae559093d792c187f0d89683f8abfc80e7dd4f103c0636200aacc0e28d WatchSource:0}: Error finding container 159ee1ae559093d792c187f0d89683f8abfc80e7dd4f103c0636200aacc0e28d: Status 404 returned error can't find the container with id 159ee1ae559093d792c187f0d89683f8abfc80e7dd4f103c0636200aacc0e28d Oct 30 12:33:10.423711 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:10.423685 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns"] Oct 30 12:33:10.497543 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:33:10.497506 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd2c069_88b7_4611_b517_c836547c29f2.slice/crio-040ea9f5ff1390788c7b380374f1cb81c574fd800bb44c59e1dda6c91ac66ffd WatchSource:0}: Error finding container 040ea9f5ff1390788c7b380374f1cb81c574fd800bb44c59e1dda6c91ac66ffd: Status 404 returned error can't find the container with id 040ea9f5ff1390788c7b380374f1cb81c574fd800bb44c59e1dda6c91ac66ffd Oct 30 12:33:10.499976 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:10.499950 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp"] Oct 30 12:33:11.266530 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:11.266484 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp" event={"ID":"8cd2c069-88b7-4611-b517-c836547c29f2","Type":"ContainerStarted","Data":"dacb57f803f60001fd33bbde4c9c095156b28341c4ee57532a30ea754241a428"} Oct 30 12:33:11.266756 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:11.266555 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp" event={"ID":"8cd2c069-88b7-4611-b517-c836547c29f2","Type":"ContainerStarted","Data":"040ea9f5ff1390788c7b380374f1cb81c574fd800bb44c59e1dda6c91ac66ffd"} Oct 30 12:33:11.266756 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:11.266677 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp" Oct 30 12:33:11.268112 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:11.268061 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns" event={"ID":"4b5e5158-cd9b-4722-bd22-b0ab3f3e299d","Type":"ContainerStarted","Data":"8f560a26dd8ec869793d8907acbf57b060d2a34e6bd8488d23d8aad3d1289cb4"} Oct 30 12:33:11.268245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:11.268129 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns" Oct 30 12:33:11.268245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:11.268145 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns" event={"ID":"4b5e5158-cd9b-4722-bd22-b0ab3f3e299d","Type":"ContainerStarted","Data":"159ee1ae559093d792c187f0d89683f8abfc80e7dd4f103c0636200aacc0e28d"} Oct 30 12:33:11.294592 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:11.294555 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" podUID="b36f2ed4-3517-4110-ab7a-8cfae8243bab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:33:11.310239 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:11.310184 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp" podStartSLOduration=2.310168729 podStartE2EDuration="2.310168729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:33:11.308888735 +0000 UTC m=+2342.343027125" watchObservedRunningTime="2026-10-30 12:33:11.310168729 +0000 UTC m=+2342.344307124" Oct 30 12:33:11.374519 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:11.374471 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns" podStartSLOduration=2.374456343 podStartE2EDuration="2.374456343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:33:11.369755905 +0000 UTC m=+2342.403894302" watchObservedRunningTime="2026-10-30 12:33:11.374456343 +0000 UTC m=+2342.408594739" Oct 30 12:33:14.948331 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:14.948299 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" Oct 30 12:33:15.026175 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:15.026141 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b36f2ed4-3517-4110-ab7a-8cfae8243bab-kube-api-access-4tmjz\" (UniqueName: \"kubernetes.io/projected/b36f2ed4-3517-4110-ab7a-8cfae8243bab-kube-api-access-4tmjz\") pod \"b36f2ed4-3517-4110-ab7a-8cfae8243bab\" (UID: \"b36f2ed4-3517-4110-ab7a-8cfae8243bab\") " Oct 30 12:33:15.026400 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:15.026218 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b36f2ed4-3517-4110-ab7a-8cfae8243bab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b36f2ed4-3517-4110-ab7a-8cfae8243bab-plugins-config-volume\") pod \"b36f2ed4-3517-4110-ab7a-8cfae8243bab\" (UID: \"b36f2ed4-3517-4110-ab7a-8cfae8243bab\") " Oct 30 12:33:15.026658 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:15.026623 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b36f2ed4-3517-4110-ab7a-8cfae8243bab-plugins-config-volume" pod "b36f2ed4-3517-4110-ab7a-8cfae8243bab" (UID: "b36f2ed4-3517-4110-ab7a-8cfae8243bab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:15.026900 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:15.026878 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b36f2ed4-3517-4110-ab7a-8cfae8243bab-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:33:15.029041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:15.029011 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36f2ed4-3517-4110-ab7a-8cfae8243bab-kube-api-access-4tmjz" pod "b36f2ed4-3517-4110-ab7a-8cfae8243bab" (UID: "b36f2ed4-3517-4110-ab7a-8cfae8243bab"). InnerVolumeSpecName "kube-api-access-4tmjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:15.128440 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:15.128399 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tmjz\" (UniqueName: \"kubernetes.io/projected/b36f2ed4-3517-4110-ab7a-8cfae8243bab-kube-api-access-4tmjz\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:33:15.293588 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:15.293501 4468 generic.go:356] "Generic (PLEG): container finished" podID="b36f2ed4-3517-4110-ab7a-8cfae8243bab" containerID="75a7c0baa2537a337a615134ca9d593d22b8c3ab8f9d6d4acf6cad38922e1a5c" exitCode=0 Oct 30 12:33:15.293588 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:15.293552 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" Oct 30 12:33:15.293777 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:15.293587 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" event={"ID":"b36f2ed4-3517-4110-ab7a-8cfae8243bab","Type":"ContainerDied","Data":"75a7c0baa2537a337a615134ca9d593d22b8c3ab8f9d6d4acf6cad38922e1a5c"} Oct 30 12:33:15.293777 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:15.293648 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr" event={"ID":"b36f2ed4-3517-4110-ab7a-8cfae8243bab","Type":"ContainerDied","Data":"ba3a17e83d0c19b92f8bea2d3650d37d31ef6cf5c2cce5829f07825ede77ab61"} Oct 30 12:33:15.293777 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:15.293674 4468 scope.go:122] "RemoveContainer" containerID="75a7c0baa2537a337a615134ca9d593d22b8c3ab8f9d6d4acf6cad38922e1a5c" Oct 30 12:33:15.303713 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:15.303694 4468 scope.go:122] "RemoveContainer" containerID="75a7c0baa2537a337a615134ca9d593d22b8c3ab8f9d6d4acf6cad38922e1a5c" Oct 30 12:33:15.304029 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:33:15.304006 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a7c0baa2537a337a615134ca9d593d22b8c3ab8f9d6d4acf6cad38922e1a5c\": container with ID starting with 75a7c0baa2537a337a615134ca9d593d22b8c3ab8f9d6d4acf6cad38922e1a5c not found: ID does not exist" containerID="75a7c0baa2537a337a615134ca9d593d22b8c3ab8f9d6d4acf6cad38922e1a5c" Oct 30 12:33:15.304135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:15.304037 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a7c0baa2537a337a615134ca9d593d22b8c3ab8f9d6d4acf6cad38922e1a5c"} err="failed to get container status \"75a7c0baa2537a337a615134ca9d593d22b8c3ab8f9d6d4acf6cad38922e1a5c\": rpc error: code = NotFound desc = could not find container \"75a7c0baa2537a337a615134ca9d593d22b8c3ab8f9d6d4acf6cad38922e1a5c\": container with ID starting with 75a7c0baa2537a337a615134ca9d593d22b8c3ab8f9d6d4acf6cad38922e1a5c not found: ID does not exist" Oct 30 12:33:15.352940 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:15.352906 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr"] Oct 30 12:33:15.365998 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:15.365969 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c44eee-5f6877c847-gwhmr"] Oct 30 12:33:15.549171 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:15.549093 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7"] Oct 30 12:33:15.549369 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:15.549343 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" podUID="04fb37c3-e5de-408c-bb27-88502637597e" containerName="maas-api" containerID="cri-o://d30421e56879d6062631d7d4b42cda0e9fbdbac025c7b4ad66cd99af37bc7f82" gracePeriod=30 Oct 30 12:33:16.220126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.220035 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" Oct 30 12:33:16.276679 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.276640 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04fb37c3-e5de-408c-bb27-88502637597e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04fb37c3-e5de-408c-bb27-88502637597e-metrics-certs\") pod \"04fb37c3-e5de-408c-bb27-88502637597e\" (UID: \"04fb37c3-e5de-408c-bb27-88502637597e\") " Oct 30 12:33:16.276900 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.276802 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fb37c3-e5de-408c-bb27-88502637597e-kube-api-access-dnfs5\" (UniqueName: \"kubernetes.io/projected/04fb37c3-e5de-408c-bb27-88502637597e-kube-api-access-dnfs5\") pod \"04fb37c3-e5de-408c-bb27-88502637597e\" (UID: \"04fb37c3-e5de-408c-bb27-88502637597e\") " Oct 30 12:33:16.276900 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.276880 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04fb37c3-e5de-408c-bb27-88502637597e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04fb37c3-e5de-408c-bb27-88502637597e-maas-api-tls\") pod \"04fb37c3-e5de-408c-bb27-88502637597e\" (UID: \"04fb37c3-e5de-408c-bb27-88502637597e\") " Oct 30 12:33:16.277051 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.276924 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04fb37c3-e5de-408c-bb27-88502637597e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04fb37c3-e5de-408c-bb27-88502637597e-openshift-service-ca\") pod \"04fb37c3-e5de-408c-bb27-88502637597e\" (UID: \"04fb37c3-e5de-408c-bb27-88502637597e\") " Oct 30 12:33:16.277537 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.277501 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04fb37c3-e5de-408c-bb27-88502637597e-openshift-service-ca" pod "04fb37c3-e5de-408c-bb27-88502637597e" (UID: "04fb37c3-e5de-408c-bb27-88502637597e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:16.279849 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.279821 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04fb37c3-e5de-408c-bb27-88502637597e-metrics-certs" pod "04fb37c3-e5de-408c-bb27-88502637597e" (UID: "04fb37c3-e5de-408c-bb27-88502637597e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:16.279958 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.279862 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fb37c3-e5de-408c-bb27-88502637597e-kube-api-access-dnfs5" pod "04fb37c3-e5de-408c-bb27-88502637597e" (UID: "04fb37c3-e5de-408c-bb27-88502637597e"). InnerVolumeSpecName "kube-api-access-dnfs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:16.280190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.280160 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04fb37c3-e5de-408c-bb27-88502637597e-maas-api-tls" pod "04fb37c3-e5de-408c-bb27-88502637597e" (UID: "04fb37c3-e5de-408c-bb27-88502637597e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:16.304587 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.304550 4468 generic.go:356] "Generic (PLEG): container finished" podID="04fb37c3-e5de-408c-bb27-88502637597e" containerID="d30421e56879d6062631d7d4b42cda0e9fbdbac025c7b4ad66cd99af37bc7f82" exitCode=0 Oct 30 12:33:16.305096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.304651 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" Oct 30 12:33:16.305096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.304657 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" event={"ID":"04fb37c3-e5de-408c-bb27-88502637597e","Type":"ContainerDied","Data":"d30421e56879d6062631d7d4b42cda0e9fbdbac025c7b4ad66cd99af37bc7f82"} Oct 30 12:33:16.305096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.304782 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7" event={"ID":"04fb37c3-e5de-408c-bb27-88502637597e","Type":"ContainerDied","Data":"74a6c56fe0ae6632d2ed7e844c7e27297ac7760e30258358ba4f2b4aa124faca"} Oct 30 12:33:16.305096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.304823 4468 scope.go:122] "RemoveContainer" containerID="d30421e56879d6062631d7d4b42cda0e9fbdbac025c7b4ad66cd99af37bc7f82" Oct 30 12:33:16.327789 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.327764 4468 scope.go:122] "RemoveContainer" containerID="d30421e56879d6062631d7d4b42cda0e9fbdbac025c7b4ad66cd99af37bc7f82" Oct 30 12:33:16.328165 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:33:16.328142 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30421e56879d6062631d7d4b42cda0e9fbdbac025c7b4ad66cd99af37bc7f82\": container with ID starting with d30421e56879d6062631d7d4b42cda0e9fbdbac025c7b4ad66cd99af37bc7f82 not found: ID does not exist" containerID="d30421e56879d6062631d7d4b42cda0e9fbdbac025c7b4ad66cd99af37bc7f82" Oct 30 12:33:16.328231 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.328175 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30421e56879d6062631d7d4b42cda0e9fbdbac025c7b4ad66cd99af37bc7f82"} err="failed to get container status \"d30421e56879d6062631d7d4b42cda0e9fbdbac025c7b4ad66cd99af37bc7f82\": rpc error: code = NotFound desc = could not find container \"d30421e56879d6062631d7d4b42cda0e9fbdbac025c7b4ad66cd99af37bc7f82\": container with ID starting with d30421e56879d6062631d7d4b42cda0e9fbdbac025c7b4ad66cd99af37bc7f82 not found: ID does not exist" Oct 30 12:33:16.367283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.367245 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7"] Oct 30 12:33:16.374117 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.374086 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c44eee-5cb74f8747-dtkn7"] Oct 30 12:33:16.379166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.379142 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnfs5\" (UniqueName: \"kubernetes.io/projected/04fb37c3-e5de-408c-bb27-88502637597e-kube-api-access-dnfs5\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:33:16.379266 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.379167 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04fb37c3-e5de-408c-bb27-88502637597e-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:33:16.379266 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.379178 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04fb37c3-e5de-408c-bb27-88502637597e-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:33:16.379266 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:16.379186 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04fb37c3-e5de-408c-bb27-88502637597e-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:33:17.090517 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:17.090484 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04fb37c3-e5de-408c-bb27-88502637597e" path="/var/lib/kubelet/pods/04fb37c3-e5de-408c-bb27-88502637597e/volumes" Oct 30 12:33:17.090949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:17.090934 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b36f2ed4-3517-4110-ab7a-8cfae8243bab" path="/var/lib/kubelet/pods/b36f2ed4-3517-4110-ab7a-8cfae8243bab/volumes" Oct 30 12:33:22.283088 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:22.281745 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp" Oct 30 12:33:22.286243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:22.285391 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns" Oct 30 12:33:23.844251 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:33:23.844208 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:33:23.989094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:23.986185 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:23.997094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:23.993153 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:24.434091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:24.432173 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 12:33:24.434091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:24.432585 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 12:33:24.434091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:24.434063 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:24.708320 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:24.695348 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 12:33:24.708320 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:24.697826 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:24.742092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:24.729037 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 12:33:24.988152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:24.987163 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:24.988152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:24.987917 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:33:25.443423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:25.436174 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:25.443423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:25.436586 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 12:33:25.443423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:25.436856 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 12:33:25.443423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:25.437491 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:33:25.443423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:25.437552 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:33:25.443423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:25.437590 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:33:25.609470 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:25.607123 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:33:25.633095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:25.628808 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:33:25.681844 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:25.669620 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:33:25.740145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:25.739660 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:25.741893 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:25.740438 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:33:25.744494 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:25.744463 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:33:25.761914 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:25.761892 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:33:26.631115 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.631061 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn"] Oct 30 12:33:26.631737 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.631719 4468 cpu_manager.go:414] "containerMap: removing container" podUID="b36f2ed4-3517-4110-ab7a-8cfae8243bab" containerName="payload-pre-processing" Oct 30 12:33:26.631815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.631738 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b36f2ed4-3517-4110-ab7a-8cfae8243bab" containerName="payload-pre-processing" Oct 30 12:33:26.631815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.631758 4468 cpu_manager.go:414] "containerMap: removing container" podUID="04fb37c3-e5de-408c-bb27-88502637597e" containerName="maas-api" Oct 30 12:33:26.631815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.631765 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04fb37c3-e5de-408c-bb27-88502637597e" containerName="maas-api" Oct 30 12:33:26.631982 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.631882 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b36f2ed4-3517-4110-ab7a-8cfae8243bab" containerName="payload-pre-processing" Oct 30 12:33:26.631982 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.631897 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="04fb37c3-e5de-408c-bb27-88502637597e" containerName="maas-api" Oct 30 12:33:26.637180 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.637159 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" Oct 30 12:33:26.656186 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.656154 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn"] Oct 30 12:33:26.656563 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.656213 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-cc08f1-serving-cert\"" Oct 30 12:33:26.800838 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.800794 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca98735-7b6b-41bc-b651-b21358aab369-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn\" (UID: \"dca98735-7b6b-41bc-b651-b21358aab369\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" Oct 30 12:33:26.801118 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.801032 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca98735-7b6b-41bc-b651-b21358aab369-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn\" (UID: \"dca98735-7b6b-41bc-b651-b21358aab369\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" Oct 30 12:33:26.801118 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.801100 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca98735-7b6b-41bc-b651-b21358aab369-metrics-certs\") pod \"maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn\" (UID: \"dca98735-7b6b-41bc-b651-b21358aab369\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" Oct 30 12:33:26.801322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.801307 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4xts\" (UniqueName: \"kubernetes.io/projected/dca98735-7b6b-41bc-b651-b21358aab369-kube-api-access-b4xts\") pod \"maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn\" (UID: \"dca98735-7b6b-41bc-b651-b21358aab369\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" Oct 30 12:33:26.903382 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.903144 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca98735-7b6b-41bc-b651-b21358aab369-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn\" (UID: \"dca98735-7b6b-41bc-b651-b21358aab369\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" Oct 30 12:33:26.903382 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.903235 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca98735-7b6b-41bc-b651-b21358aab369-metrics-certs\") pod \"maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn\" (UID: \"dca98735-7b6b-41bc-b651-b21358aab369\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" Oct 30 12:33:26.903728 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.903388 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xts\" (UniqueName: \"kubernetes.io/projected/dca98735-7b6b-41bc-b651-b21358aab369-kube-api-access-b4xts\") pod \"maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn\" (UID: \"dca98735-7b6b-41bc-b651-b21358aab369\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" Oct 30 12:33:26.903728 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.903450 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca98735-7b6b-41bc-b651-b21358aab369-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn\" (UID: \"dca98735-7b6b-41bc-b651-b21358aab369\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" Oct 30 12:33:26.904934 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.904905 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca98735-7b6b-41bc-b651-b21358aab369-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn\" (UID: \"dca98735-7b6b-41bc-b651-b21358aab369\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" Oct 30 12:33:26.924876 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.924823 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca98735-7b6b-41bc-b651-b21358aab369-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn\" (UID: \"dca98735-7b6b-41bc-b651-b21358aab369\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" Oct 30 12:33:26.926929 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.926901 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca98735-7b6b-41bc-b651-b21358aab369-metrics-certs\") pod \"maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn\" (UID: \"dca98735-7b6b-41bc-b651-b21358aab369\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" Oct 30 12:33:26.931832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.931802 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4xts\" (UniqueName: \"kubernetes.io/projected/dca98735-7b6b-41bc-b651-b21358aab369-kube-api-access-b4xts\") pod \"maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn\" (UID: \"dca98735-7b6b-41bc-b651-b21358aab369\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" Oct 30 12:33:26.958551 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:26.958521 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" Oct 30 12:33:27.279983 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:33:27.279948 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca98735_7b6b_41bc_b651_b21358aab369.slice/crio-a94e50b9d33b62da17a2e9d0488f703e20af66f9face4292dfcf106b712ccc12 WatchSource:0}: Error finding container a94e50b9d33b62da17a2e9d0488f703e20af66f9face4292dfcf106b712ccc12: Status 404 returned error can't find the container with id a94e50b9d33b62da17a2e9d0488f703e20af66f9face4292dfcf106b712ccc12 Oct 30 12:33:27.283109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:27.283064 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn"] Oct 30 12:33:27.727481 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:27.727451 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" event={"ID":"dca98735-7b6b-41bc-b651-b21358aab369","Type":"ContainerStarted","Data":"a94e50b9d33b62da17a2e9d0488f703e20af66f9face4292dfcf106b712ccc12"} Oct 30 12:33:28.736409 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:28.736374 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" event={"ID":"dca98735-7b6b-41bc-b651-b21358aab369","Type":"ContainerStarted","Data":"57c71c5a6275d22b1935a177f4e2755c6176cb9d267d2450c046a0f2bcabc685"} Oct 30 12:33:28.736875 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:28.736457 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" Oct 30 12:33:28.794859 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:28.794790 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" podStartSLOduration=2.417836951 podStartE2EDuration="2.794766211s" totalImagesPullingTime="376.92926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:33:28.792506171 +0000 UTC m=+2359.826644568" watchObservedRunningTime="2026-10-30 12:33:28.794766211 +0000 UTC m=+2359.828904607" Oct 30 12:33:34.750086 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:34.750047 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" Oct 30 12:33:41.379094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:41.367570 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:41.629153 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:41.612236 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 12:33:41.655095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:41.637169 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:41.655095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:41.637612 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 12:33:41.675224 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:41.675167 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:41.690169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:41.690125 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 12:33:41.699206 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:41.699153 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:41.760219 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:41.760168 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:41.768727 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:41.768678 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:41.994425 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:41.993175 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:42.631095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:42.616167 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:42.631095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:42.617009 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:33:42.639100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:42.638884 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 12:33:42.651137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:42.639378 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:33:42.726095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:42.698293 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:42.726095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:42.699087 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:33:42.726095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:42.700047 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:42.726095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:42.700442 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:33:42.787116 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:42.786998 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 12:33:42.796240 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:42.787475 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:33:42.912183 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:42.905248 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:42.922093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:42.906190 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:33:42.922093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:42.919268 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 12:33:42.922093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:42.919683 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:33:43.105679 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:43.104476 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:43.272147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:43.270256 4468 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:33:43.272147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:43.270327 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" podUID="ea87399a-8203-4239-a086-097a4bb9120e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:43.612142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:43.611415 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded" Oct 30 12:33:43.624516 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:43.618681 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 12:33:43.646090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:43.641251 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:43.699398 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:43.699096 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:43.725095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:43.710740 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:43.725095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:43.713631 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:33:43.952108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:43.949439 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 12:33:43.960113 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:43.960000 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 12:33:44.267033 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:44.266938 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:44.294098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:44.265339 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 12:33:44.294098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:44.273403 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:33:44.294098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:44.292286 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:33:44.294098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:44.292386 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:33:44.294098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:44.292465 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:33:44.312244 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:44.297052 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:33:44.312244 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:44.297150 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:33:44.376115 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:44.374463 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:33:44.383703 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:44.381952 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:33:51.387552 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:51.387470 4468 generic.go:356] "Generic (PLEG): container finished" podID="f9f83c6f-04b8-46a1-824a-da232fe7c23d" containerID="c0fa5408a45f5b4d0b5d12f2afb6f025f883d5dff83297a912598d0734f468cf" exitCode=0 Oct 30 12:33:51.387968 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:51.387547 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" event={"ID":"f9f83c6f-04b8-46a1-824a-da232fe7c23d","Type":"ContainerDied","Data":"c0fa5408a45f5b4d0b5d12f2afb6f025f883d5dff83297a912598d0734f468cf"} Oct 30 12:33:51.388018 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:51.387977 4468 scope.go:122] "RemoveContainer" containerID="c0fa5408a45f5b4d0b5d12f2afb6f025f883d5dff83297a912598d0734f468cf" Oct 30 12:33:51.389475 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:51.389451 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:33:51.389539 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:51.389516 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:33:51.389581 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:51.389548 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:33:52.235379 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:52.235344 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:52.397060 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:52.397022 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" event={"ID":"f9f83c6f-04b8-46a1-824a-da232fe7c23d","Type":"ContainerStarted","Data":"ae493a849ba07886a3112b65006e7d4427a8b80df1f2ec259768476b8cc489a2"} Oct 30 12:33:52.658657 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:52.658624 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:52.659945 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:52.659918 4468 patch_prober.go:29] interesting pod/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 12:33:52.660065 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:52.659970 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" podUID="f9f83c6f-04b8-46a1-824a-da232fe7c23d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 12:33:53.600151 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:33:53.600109 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:33:53.658306 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:53.658266 4468 patch_prober.go:29] interesting pod/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 12:33:53.658481 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:53.658323 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" podUID="f9f83c6f-04b8-46a1-824a-da232fe7c23d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 12:33:54.589571 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:54.589529 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl"] Oct 30 12:33:54.658038 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:54.658003 4468 patch_prober.go:29] interesting pod/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 12:33:54.658665 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:54.658631 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" podUID="f9f83c6f-04b8-46a1-824a-da232fe7c23d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 12:33:55.658274 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:55.658237 4468 patch_prober.go:29] interesting pod/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 12:33:55.658714 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:55.658301 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" podUID="f9f83c6f-04b8-46a1-824a-da232fe7c23d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 12:33:56.658433 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:56.658391 4468 patch_prober.go:29] interesting pod/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 12:33:56.658954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:56.658471 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" podUID="f9f83c6f-04b8-46a1-824a-da232fe7c23d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 12:33:57.662953 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:57.662922 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:57.663472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:57.663450 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:57.664374 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:57.664321 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:33:59.590657 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:59.590611 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl" podUID="efb3943e-edd9-4e48-affc-17631ef96950" containerName="payload-pre-processing" containerID="cri-o://8739320b8913d37dd5d073d29b82a49a9c10d8d4189cee92a7f6686a240cdb1d" gracePeriod=25 Oct 30 12:33:59.895011 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:59.894980 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl" Oct 30 12:33:59.936646 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:59.935611 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb3943e-edd9-4e48-affc-17631ef96950-kube-api-access-ns2j7\" (UniqueName: \"kubernetes.io/projected/efb3943e-edd9-4e48-affc-17631ef96950-kube-api-access-ns2j7\") pod \"efb3943e-edd9-4e48-affc-17631ef96950\" (UID: \"efb3943e-edd9-4e48-affc-17631ef96950\") " Oct 30 12:33:59.936646 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:59.935728 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efb3943e-edd9-4e48-affc-17631ef96950-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efb3943e-edd9-4e48-affc-17631ef96950-plugins-config-volume\") pod \"efb3943e-edd9-4e48-affc-17631ef96950\" (UID: \"efb3943e-edd9-4e48-affc-17631ef96950\") " Oct 30 12:33:59.937291 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:59.937258 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb3943e-edd9-4e48-affc-17631ef96950-plugins-config-volume" pod "efb3943e-edd9-4e48-affc-17631ef96950" (UID: "efb3943e-edd9-4e48-affc-17631ef96950"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:59.939162 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:33:59.939133 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb3943e-edd9-4e48-affc-17631ef96950-kube-api-access-ns2j7" pod "efb3943e-edd9-4e48-affc-17631ef96950" (UID: "efb3943e-edd9-4e48-affc-17631ef96950"). InnerVolumeSpecName "kube-api-access-ns2j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:00.038913 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:00.038875 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns2j7\" (UniqueName: \"kubernetes.io/projected/efb3943e-edd9-4e48-affc-17631ef96950-kube-api-access-ns2j7\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:00.038913 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:00.038908 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efb3943e-edd9-4e48-affc-17631ef96950-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:00.492226 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:00.492193 4468 generic.go:356] "Generic (PLEG): container finished" podID="efb3943e-edd9-4e48-affc-17631ef96950" containerID="8739320b8913d37dd5d073d29b82a49a9c10d8d4189cee92a7f6686a240cdb1d" exitCode=0 Oct 30 12:34:00.492423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:00.492237 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl" Oct 30 12:34:00.492423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:00.492270 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl" event={"ID":"efb3943e-edd9-4e48-affc-17631ef96950","Type":"ContainerDied","Data":"8739320b8913d37dd5d073d29b82a49a9c10d8d4189cee92a7f6686a240cdb1d"} Oct 30 12:34:00.492423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:00.492324 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl" event={"ID":"efb3943e-edd9-4e48-affc-17631ef96950","Type":"ContainerDied","Data":"ee7d46af8149fcd1459a7ea538d6e3115f91a4d8b4f144e8de5e28f6d6ff4247"} Oct 30 12:34:00.492423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:00.492345 4468 scope.go:122] "RemoveContainer" containerID="8739320b8913d37dd5d073d29b82a49a9c10d8d4189cee92a7f6686a240cdb1d" Oct 30 12:34:00.507445 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:00.507419 4468 scope.go:122] "RemoveContainer" containerID="8739320b8913d37dd5d073d29b82a49a9c10d8d4189cee92a7f6686a240cdb1d" Oct 30 12:34:00.507757 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:34:00.507738 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8739320b8913d37dd5d073d29b82a49a9c10d8d4189cee92a7f6686a240cdb1d\": container with ID starting with 8739320b8913d37dd5d073d29b82a49a9c10d8d4189cee92a7f6686a240cdb1d not found: ID does not exist" containerID="8739320b8913d37dd5d073d29b82a49a9c10d8d4189cee92a7f6686a240cdb1d" Oct 30 12:34:00.507849 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:00.507767 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8739320b8913d37dd5d073d29b82a49a9c10d8d4189cee92a7f6686a240cdb1d"} err="failed to get container status \"8739320b8913d37dd5d073d29b82a49a9c10d8d4189cee92a7f6686a240cdb1d\": rpc error: code = NotFound desc = could not find container \"8739320b8913d37dd5d073d29b82a49a9c10d8d4189cee92a7f6686a240cdb1d\": container with ID starting with 8739320b8913d37dd5d073d29b82a49a9c10d8d4189cee92a7f6686a240cdb1d not found: ID does not exist" Oct 30 12:34:00.609724 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:00.609688 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl"] Oct 30 12:34:00.622358 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:00.622312 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2461e8-846d9fcf67-27mbl"] Oct 30 12:34:01.093152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:01.093094 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efb3943e-edd9-4e48-affc-17631ef96950" path="/var/lib/kubelet/pods/efb3943e-edd9-4e48-affc-17631ef96950/volumes" Oct 30 12:34:04.327093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:04.324877 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:04.364261 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:04.364205 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:04.725232 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:04.725181 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:04.974108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:04.964178 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:05.287095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:05.279177 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:05.287095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:05.279604 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 12:34:05.325252 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:05.325194 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:05.325922 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:05.325891 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 12:34:05.327430 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:05.326619 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:34:05.409145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:05.392555 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 12:34:05.415715 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:05.415680 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:34:05.654320 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:05.654174 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:05.896094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:05.836750 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 30 12:34:05.896094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:05.837103 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:34:05.968859 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:05.968810 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:06.013105 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:05.970257 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:34:06.280288 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:06.280183 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:34:06.280544 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:06.280514 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:06.282559 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:06.280999 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:34:06.282559 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:06.280514 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:06.283875 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:06.283852 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:34:06.358395 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:06.347266 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:06.372195 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:06.365489 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:06.372195 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:06.371174 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:34:06.423123 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:06.420505 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:06.830000 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:06.829967 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:34:06.837837 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:06.837792 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:06.854364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:06.844645 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj"] Oct 30 12:34:06.982556 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:06.982522 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:34:07.018224 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:07.018103 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:34:07.019771 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:07.018705 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:34:07.185127 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:07.184644 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:34:07.205807 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:07.203038 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:34:11.845268 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:11.845209 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj" podUID="81958156-cfb9-4398-a8a3-ca72db1036df" containerName="payload-processing" containerID="cri-o://e44b720363568dbf011c0226383474f63489c8d0b4452b198d04a578dae4d6c9" gracePeriod=25 Oct 30 12:34:12.121780 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:12.121750 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj" Oct 30 12:34:12.176094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:12.174171 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81958156-cfb9-4398-a8a3-ca72db1036df-kube-api-access-f67nn\" (UniqueName: \"kubernetes.io/projected/81958156-cfb9-4398-a8a3-ca72db1036df-kube-api-access-f67nn\") pod \"81958156-cfb9-4398-a8a3-ca72db1036df\" (UID: \"81958156-cfb9-4398-a8a3-ca72db1036df\") " Oct 30 12:34:12.176094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:12.174311 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81958156-cfb9-4398-a8a3-ca72db1036df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81958156-cfb9-4398-a8a3-ca72db1036df-plugins-config-volume\") pod \"81958156-cfb9-4398-a8a3-ca72db1036df\" (UID: \"81958156-cfb9-4398-a8a3-ca72db1036df\") " Oct 30 12:34:12.179389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:12.178792 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81958156-cfb9-4398-a8a3-ca72db1036df-plugins-config-volume" pod "81958156-cfb9-4398-a8a3-ca72db1036df" (UID: "81958156-cfb9-4398-a8a3-ca72db1036df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:12.213669 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:12.213631 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81958156-cfb9-4398-a8a3-ca72db1036df-kube-api-access-f67nn" pod "81958156-cfb9-4398-a8a3-ca72db1036df" (UID: "81958156-cfb9-4398-a8a3-ca72db1036df"). InnerVolumeSpecName "kube-api-access-f67nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:12.276056 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:12.276020 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81958156-cfb9-4398-a8a3-ca72db1036df-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:12.276056 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:12.276057 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f67nn\" (UniqueName: \"kubernetes.io/projected/81958156-cfb9-4398-a8a3-ca72db1036df-kube-api-access-f67nn\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:12.558017 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:12.557981 4468 generic.go:356] "Generic (PLEG): container finished" podID="81958156-cfb9-4398-a8a3-ca72db1036df" containerID="e44b720363568dbf011c0226383474f63489c8d0b4452b198d04a578dae4d6c9" exitCode=0 Oct 30 12:34:12.558267 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:12.558029 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj" Oct 30 12:34:12.558267 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:12.558089 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj" event={"ID":"81958156-cfb9-4398-a8a3-ca72db1036df","Type":"ContainerDied","Data":"e44b720363568dbf011c0226383474f63489c8d0b4452b198d04a578dae4d6c9"} Oct 30 12:34:12.558267 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:12.558149 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj" event={"ID":"81958156-cfb9-4398-a8a3-ca72db1036df","Type":"ContainerDied","Data":"68dc115ef9dc6f3cd21a07c346d5711aec25d238e8329d5872067a75513898b0"} Oct 30 12:34:12.558267 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:12.558173 4468 scope.go:122] "RemoveContainer" containerID="e44b720363568dbf011c0226383474f63489c8d0b4452b198d04a578dae4d6c9" Oct 30 12:34:12.568571 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:12.568551 4468 scope.go:122] "RemoveContainer" containerID="e44b720363568dbf011c0226383474f63489c8d0b4452b198d04a578dae4d6c9" Oct 30 12:34:12.568893 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:34:12.568873 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e44b720363568dbf011c0226383474f63489c8d0b4452b198d04a578dae4d6c9\": container with ID starting with e44b720363568dbf011c0226383474f63489c8d0b4452b198d04a578dae4d6c9 not found: ID does not exist" containerID="e44b720363568dbf011c0226383474f63489c8d0b4452b198d04a578dae4d6c9" Oct 30 12:34:12.568964 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:12.568906 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e44b720363568dbf011c0226383474f63489c8d0b4452b198d04a578dae4d6c9"} err="failed to get container status \"e44b720363568dbf011c0226383474f63489c8d0b4452b198d04a578dae4d6c9\": rpc error: code = NotFound desc = could not find container \"e44b720363568dbf011c0226383474f63489c8d0b4452b198d04a578dae4d6c9\": container with ID starting with e44b720363568dbf011c0226383474f63489c8d0b4452b198d04a578dae4d6c9 not found: ID does not exist" Oct 30 12:34:12.617701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:12.617669 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj"] Oct 30 12:34:12.631107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:12.631057 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a124ab-66468fdbc8-v4bfj"] Oct 30 12:34:13.090687 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:13.090652 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81958156-cfb9-4398-a8a3-ca72db1036df" path="/var/lib/kubelet/pods/81958156-cfb9-4398-a8a3-ca72db1036df/volumes" Oct 30 12:34:18.173007 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:18.172962 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:18.209375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:18.208645 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 12:34:18.235094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:18.234498 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 12:34:18.638296 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:18.638182 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:18.682368 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:18.682191 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:34:18.682368 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:18.682260 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" podUID="e286532f-d8eb-4c7f-baea-7c71dab94e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:18.851095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:18.848304 4468 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:34:18.851095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:18.848378 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" podUID="bdc9d563-a787-4dac-b192-9507eaad1b14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:18.876096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:18.865189 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:18.876096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:18.865578 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 12:34:19.024742 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:19.000300 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:19.049094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:19.030457 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:19.185968 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:19.177428 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:19.218102 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:19.215735 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:19.246127 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:19.242090 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:19.641103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:19.640551 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 12:34:19.821093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:19.820483 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:34:19.821093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:19.820572 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:34:19.821093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:19.820622 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:34:19.821093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:19.820669 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:34:19.880093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:19.875082 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:19.880093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:19.875212 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 30 12:34:20.061092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:20.038325 4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded" Oct 30 12:34:20.061092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:20.049954 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:34:20.061092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:20.053610 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:34:20.061092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:20.051732 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:20.061092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:20.052712 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:20.108943 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:20.075298 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:34:20.108943 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:20.075447 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:34:20.249199 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:20.237015 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:34:20.319615 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:20.319506 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 12:34:20.384121 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:20.358697 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:20.384121 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:20.377714 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:34:20.384121 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:20.378568 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:34:20.429870 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:20.429839 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:34:20.448891 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:20.448831 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:34:20.450417 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:20.449994 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:34:20.465821 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:20.453640 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:34:20.470892 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:20.469103 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:34:22.324389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:22.324349 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj"] Oct 30 12:34:23.599653 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:34:23.599618 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:34:27.324790 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:27.324741 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj" podUID="aa3ba54c-2c7d-4274-97a4-f6335649d9e7" containerName="payload-pre-processing" containerID="cri-o://56a51600ff853a1afb777a8417a4456ce7427273da2d62cf6ff23437006bb64e" gracePeriod=25 Oct 30 12:34:27.651543 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:27.651512 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj" Oct 30 12:34:27.763107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:27.756323 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa3ba54c-2c7d-4274-97a4-f6335649d9e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3ba54c-2c7d-4274-97a4-f6335649d9e7-plugins-config-volume\") pod \"aa3ba54c-2c7d-4274-97a4-f6335649d9e7\" (UID: \"aa3ba54c-2c7d-4274-97a4-f6335649d9e7\") " Oct 30 12:34:27.763107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:27.757012 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa3ba54c-2c7d-4274-97a4-f6335649d9e7-plugins-config-volume" pod "aa3ba54c-2c7d-4274-97a4-f6335649d9e7" (UID: "aa3ba54c-2c7d-4274-97a4-f6335649d9e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:27.763107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:27.757311 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa3ba54c-2c7d-4274-97a4-f6335649d9e7-kube-api-access-8zv2h\" (UniqueName: \"kubernetes.io/projected/aa3ba54c-2c7d-4274-97a4-f6335649d9e7-kube-api-access-8zv2h\") pod \"aa3ba54c-2c7d-4274-97a4-f6335649d9e7\" (UID: \"aa3ba54c-2c7d-4274-97a4-f6335649d9e7\") " Oct 30 12:34:27.763107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:27.759061 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3ba54c-2c7d-4274-97a4-f6335649d9e7-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:27.772571 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:27.772518 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3ba54c-2c7d-4274-97a4-f6335649d9e7-kube-api-access-8zv2h" pod "aa3ba54c-2c7d-4274-97a4-f6335649d9e7" (UID: "aa3ba54c-2c7d-4274-97a4-f6335649d9e7"). InnerVolumeSpecName "kube-api-access-8zv2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:27.860871 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:27.860831 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zv2h\" (UniqueName: \"kubernetes.io/projected/aa3ba54c-2c7d-4274-97a4-f6335649d9e7-kube-api-access-8zv2h\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:28.274198 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:28.274167 4468 generic.go:356] "Generic (PLEG): container finished" podID="aa3ba54c-2c7d-4274-97a4-f6335649d9e7" containerID="56a51600ff853a1afb777a8417a4456ce7427273da2d62cf6ff23437006bb64e" exitCode=0 Oct 30 12:34:28.274386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:28.274206 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj" Oct 30 12:34:28.274386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:28.274235 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj" event={"ID":"aa3ba54c-2c7d-4274-97a4-f6335649d9e7","Type":"ContainerDied","Data":"56a51600ff853a1afb777a8417a4456ce7427273da2d62cf6ff23437006bb64e"} Oct 30 12:34:28.274386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:28.274290 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj" event={"ID":"aa3ba54c-2c7d-4274-97a4-f6335649d9e7","Type":"ContainerDied","Data":"21ca8a0eb5f42bdbf5ff91d78312d4235c1f5c79fc4ab4bc8f30353ecfbdb414"} Oct 30 12:34:28.274386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:28.274314 4468 scope.go:122] "RemoveContainer" containerID="56a51600ff853a1afb777a8417a4456ce7427273da2d62cf6ff23437006bb64e" Oct 30 12:34:28.285864 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:28.285846 4468 scope.go:122] "RemoveContainer" containerID="56a51600ff853a1afb777a8417a4456ce7427273da2d62cf6ff23437006bb64e" Oct 30 12:34:28.286169 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:34:28.286150 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56a51600ff853a1afb777a8417a4456ce7427273da2d62cf6ff23437006bb64e\": container with ID starting with 56a51600ff853a1afb777a8417a4456ce7427273da2d62cf6ff23437006bb64e not found: ID does not exist" containerID="56a51600ff853a1afb777a8417a4456ce7427273da2d62cf6ff23437006bb64e" Oct 30 12:34:28.286250 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:28.286176 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56a51600ff853a1afb777a8417a4456ce7427273da2d62cf6ff23437006bb64e"} err="failed to get container status \"56a51600ff853a1afb777a8417a4456ce7427273da2d62cf6ff23437006bb64e\": rpc error: code = NotFound desc = could not find container \"56a51600ff853a1afb777a8417a4456ce7427273da2d62cf6ff23437006bb64e\": container with ID starting with 56a51600ff853a1afb777a8417a4456ce7427273da2d62cf6ff23437006bb64e not found: ID does not exist" Oct 30 12:34:28.332122 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:28.332089 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj"] Oct 30 12:34:28.345203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:28.345176 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5ce7c4-d4b4d65d7-ljkgj"] Oct 30 12:34:28.420122 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:28.414156 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2"] Oct 30 12:34:28.420122 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:28.414844 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" podUID="43df4a6b-2b51-4575-aff8-530380e15466" containerName="istio-proxy" containerID="cri-o://4368dfa756e10a9f825b2e1ce7e6633f35c9b48cc70947cde7312dc1c64caa0f" gracePeriod=30 Oct 30 12:34:29.090361 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:29.090325 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa3ba54c-2c7d-4274-97a4-f6335649d9e7" path="/var/lib/kubelet/pods/aa3ba54c-2c7d-4274-97a4-f6335649d9e7/volumes" Oct 30 12:34:33.698709 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.698638 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:34:33.809416 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.809383 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-istio-data\") pod \"43df4a6b-2b51-4575-aff8-530380e15466\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " Oct 30 12:34:33.809590 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.809522 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43df4a6b-2b51-4575-aff8-530380e15466-istio-token\" (UniqueName: \"kubernetes.io/projected/43df4a6b-2b51-4575-aff8-530380e15466-istio-token\") pod \"43df4a6b-2b51-4575-aff8-530380e15466\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " Oct 30 12:34:33.809640 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.809589 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-workload-socket\") pod \"43df4a6b-2b51-4575-aff8-530380e15466\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " Oct 30 12:34:33.809728 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.809676 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43df4a6b-2b51-4575-aff8-530380e15466-kube-api-access-whjwf\" (UniqueName: \"kubernetes.io/projected/43df4a6b-2b51-4575-aff8-530380e15466-kube-api-access-whjwf\") pod \"43df4a6b-2b51-4575-aff8-530380e15466\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " Oct 30 12:34:33.809728 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.809696 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-istio-data" pod "43df4a6b-2b51-4575-aff8-530380e15466" (UID: "43df4a6b-2b51-4575-aff8-530380e15466"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:33.809799 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.809727 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-credential-socket\") pod \"43df4a6b-2b51-4575-aff8-530380e15466\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " Oct 30 12:34:33.809850 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.809832 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43df4a6b-2b51-4575-aff8-530380e15466-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43df4a6b-2b51-4575-aff8-530380e15466-istiod-ca-cert\") pod \"43df4a6b-2b51-4575-aff8-530380e15466\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " Oct 30 12:34:33.809905 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.809893 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-istio-envoy\") pod \"43df4a6b-2b51-4575-aff8-530380e15466\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " Oct 30 12:34:33.809963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.809922 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-workload-socket" pod "43df4a6b-2b51-4575-aff8-530380e15466" (UID: "43df4a6b-2b51-4575-aff8-530380e15466"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:33.809963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.809932 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-credential-socket" pod "43df4a6b-2b51-4575-aff8-530380e15466" (UID: "43df4a6b-2b51-4575-aff8-530380e15466"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:33.809963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.809927 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-workload-certs\") pod \"43df4a6b-2b51-4575-aff8-530380e15466\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " Oct 30 12:34:33.810149 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.809971 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43df4a6b-2b51-4575-aff8-530380e15466-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43df4a6b-2b51-4575-aff8-530380e15466-istio-podinfo\") pod \"43df4a6b-2b51-4575-aff8-530380e15466\" (UID: \"43df4a6b-2b51-4575-aff8-530380e15466\") " Oct 30 12:34:33.810217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.810183 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-workload-certs" pod "43df4a6b-2b51-4575-aff8-530380e15466" (UID: "43df4a6b-2b51-4575-aff8-530380e15466"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:33.810310 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.810282 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43df4a6b-2b51-4575-aff8-530380e15466-istiod-ca-cert" pod "43df4a6b-2b51-4575-aff8-530380e15466" (UID: "43df4a6b-2b51-4575-aff8-530380e15466"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:33.810717 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.810691 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:33.810826 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.810722 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:33.810826 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.810737 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:33.810826 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.810754 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43df4a6b-2b51-4575-aff8-530380e15466-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:33.810826 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.810770 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:33.811997 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.811968 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43df4a6b-2b51-4575-aff8-530380e15466-istio-token" pod "43df4a6b-2b51-4575-aff8-530380e15466" (UID: "43df4a6b-2b51-4575-aff8-530380e15466"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:33.812141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.812012 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-istio-envoy" pod "43df4a6b-2b51-4575-aff8-530380e15466" (UID: "43df4a6b-2b51-4575-aff8-530380e15466"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:33.812141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.812121 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43df4a6b-2b51-4575-aff8-530380e15466-kube-api-access-whjwf" pod "43df4a6b-2b51-4575-aff8-530380e15466" (UID: "43df4a6b-2b51-4575-aff8-530380e15466"). InnerVolumeSpecName "kube-api-access-whjwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:33.812481 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.812459 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43df4a6b-2b51-4575-aff8-530380e15466-istio-podinfo" pod "43df4a6b-2b51-4575-aff8-530380e15466" (UID: "43df4a6b-2b51-4575-aff8-530380e15466"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:34:33.911844 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.911810 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whjwf\" (UniqueName: \"kubernetes.io/projected/43df4a6b-2b51-4575-aff8-530380e15466-kube-api-access-whjwf\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:33.911844 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.911841 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43df4a6b-2b51-4575-aff8-530380e15466-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:33.911844 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.911854 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43df4a6b-2b51-4575-aff8-530380e15466-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:33.912126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:33.911862 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43df4a6b-2b51-4575-aff8-530380e15466-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:34.394359 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:34.394325 4468 generic.go:356] "Generic (PLEG): container finished" podID="43df4a6b-2b51-4575-aff8-530380e15466" containerID="4368dfa756e10a9f825b2e1ce7e6633f35c9b48cc70947cde7312dc1c64caa0f" exitCode=0 Oct 30 12:34:34.394567 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:34.394505 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" event={"ID":"43df4a6b-2b51-4575-aff8-530380e15466","Type":"ContainerDied","Data":"4368dfa756e10a9f825b2e1ce7e6633f35c9b48cc70947cde7312dc1c64caa0f"} Oct 30 12:34:34.394567 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:34.394545 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" event={"ID":"43df4a6b-2b51-4575-aff8-530380e15466","Type":"ContainerDied","Data":"a5ffe9a7209ac5112ab5841e6d123d325a74490eba078288f33b7d52d511ff1f"} Oct 30 12:34:34.394567 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:34.394566 4468 scope.go:122] "RemoveContainer" containerID="4368dfa756e10a9f825b2e1ce7e6633f35c9b48cc70947cde7312dc1c64caa0f" Oct 30 12:34:34.394750 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:34.394718 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2" Oct 30 12:34:34.414714 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:34.414691 4468 scope.go:122] "RemoveContainer" containerID="4368dfa756e10a9f825b2e1ce7e6633f35c9b48cc70947cde7312dc1c64caa0f" Oct 30 12:34:34.415216 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:34:34.415049 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4368dfa756e10a9f825b2e1ce7e6633f35c9b48cc70947cde7312dc1c64caa0f\": container with ID starting with 4368dfa756e10a9f825b2e1ce7e6633f35c9b48cc70947cde7312dc1c64caa0f not found: ID does not exist" containerID="4368dfa756e10a9f825b2e1ce7e6633f35c9b48cc70947cde7312dc1c64caa0f" Oct 30 12:34:34.415216 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:34.415097 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4368dfa756e10a9f825b2e1ce7e6633f35c9b48cc70947cde7312dc1c64caa0f"} err="failed to get container status \"4368dfa756e10a9f825b2e1ce7e6633f35c9b48cc70947cde7312dc1c64caa0f\": rpc error: code = NotFound desc = could not find container \"4368dfa756e10a9f825b2e1ce7e6633f35c9b48cc70947cde7312dc1c64caa0f\": container with ID starting with 4368dfa756e10a9f825b2e1ce7e6633f35c9b48cc70947cde7312dc1c64caa0f not found: ID does not exist" Oct 30 12:34:34.535096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:34.534436 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2"] Oct 30 12:34:34.599417 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:34.598685 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-fzcd2"] Oct 30 12:34:35.040357 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.039238 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx"] Oct 30 12:34:35.041213 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.041156 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" podUID="d3162c67-d852-4c92-8d21-6f85009feaff" containerName="maas-api" containerID="cri-o://414288ee42949906afc17a0d00b0919f89335b9a5969c6fd2bc72af5be9cf3d1" gracePeriod=30 Oct 30 12:34:35.118608 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.118566 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43df4a6b-2b51-4575-aff8-530380e15466" path="/var/lib/kubelet/pods/43df4a6b-2b51-4575-aff8-530380e15466/volumes" Oct 30 12:34:35.340514 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.340486 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" Oct 30 12:34:35.351986 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.351903 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3162c67-d852-4c92-8d21-6f85009feaff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3162c67-d852-4c92-8d21-6f85009feaff-maas-api-tls\") pod \"d3162c67-d852-4c92-8d21-6f85009feaff\" (UID: \"d3162c67-d852-4c92-8d21-6f85009feaff\") " Oct 30 12:34:35.351986 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.351955 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3162c67-d852-4c92-8d21-6f85009feaff-kube-api-access-st82v\" (UniqueName: \"kubernetes.io/projected/d3162c67-d852-4c92-8d21-6f85009feaff-kube-api-access-st82v\") pod \"d3162c67-d852-4c92-8d21-6f85009feaff\" (UID: \"d3162c67-d852-4c92-8d21-6f85009feaff\") " Oct 30 12:34:35.352244 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.352126 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3162c67-d852-4c92-8d21-6f85009feaff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3162c67-d852-4c92-8d21-6f85009feaff-openshift-service-ca\") pod \"d3162c67-d852-4c92-8d21-6f85009feaff\" (UID: \"d3162c67-d852-4c92-8d21-6f85009feaff\") " Oct 30 12:34:35.352477 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.352454 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3162c67-d852-4c92-8d21-6f85009feaff-openshift-service-ca" pod "d3162c67-d852-4c92-8d21-6f85009feaff" (UID: "d3162c67-d852-4c92-8d21-6f85009feaff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:35.353051 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.353025 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3162c67-d852-4c92-8d21-6f85009feaff-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:35.354822 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.354729 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3162c67-d852-4c92-8d21-6f85009feaff-kube-api-access-st82v" pod "d3162c67-d852-4c92-8d21-6f85009feaff" (UID: "d3162c67-d852-4c92-8d21-6f85009feaff"). InnerVolumeSpecName "kube-api-access-st82v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:35.356632 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.356595 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3162c67-d852-4c92-8d21-6f85009feaff-maas-api-tls" pod "d3162c67-d852-4c92-8d21-6f85009feaff" (UID: "d3162c67-d852-4c92-8d21-6f85009feaff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:34:35.405916 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.405869 4468 generic.go:356] "Generic (PLEG): container finished" podID="d3162c67-d852-4c92-8d21-6f85009feaff" containerID="414288ee42949906afc17a0d00b0919f89335b9a5969c6fd2bc72af5be9cf3d1" exitCode=0 Oct 30 12:34:35.406141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.405936 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" Oct 30 12:34:35.406141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.405960 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" event={"ID":"d3162c67-d852-4c92-8d21-6f85009feaff","Type":"ContainerDied","Data":"414288ee42949906afc17a0d00b0919f89335b9a5969c6fd2bc72af5be9cf3d1"} Oct 30 12:34:35.406141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.406033 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx" event={"ID":"d3162c67-d852-4c92-8d21-6f85009feaff","Type":"ContainerDied","Data":"f23e31e88ac4268fca055b1cca73238f5c82c2ff5abca8c4b187849be6e9977b"} Oct 30 12:34:35.406141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.406111 4468 scope.go:122] "RemoveContainer" containerID="414288ee42949906afc17a0d00b0919f89335b9a5969c6fd2bc72af5be9cf3d1" Oct 30 12:34:35.418603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.418577 4468 scope.go:122] "RemoveContainer" containerID="414288ee42949906afc17a0d00b0919f89335b9a5969c6fd2bc72af5be9cf3d1" Oct 30 12:34:35.418925 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:34:35.418901 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414288ee42949906afc17a0d00b0919f89335b9a5969c6fd2bc72af5be9cf3d1\": container with ID starting with 414288ee42949906afc17a0d00b0919f89335b9a5969c6fd2bc72af5be9cf3d1 not found: ID does not exist" containerID="414288ee42949906afc17a0d00b0919f89335b9a5969c6fd2bc72af5be9cf3d1" Oct 30 12:34:35.419009 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.418935 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414288ee42949906afc17a0d00b0919f89335b9a5969c6fd2bc72af5be9cf3d1"} err="failed to get container status \"414288ee42949906afc17a0d00b0919f89335b9a5969c6fd2bc72af5be9cf3d1\": rpc error: code = NotFound desc = could not find container \"414288ee42949906afc17a0d00b0919f89335b9a5969c6fd2bc72af5be9cf3d1\": container with ID starting with 414288ee42949906afc17a0d00b0919f89335b9a5969c6fd2bc72af5be9cf3d1 not found: ID does not exist" Oct 30 12:34:35.454167 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.454137 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3162c67-d852-4c92-8d21-6f85009feaff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3162c67-d852-4c92-8d21-6f85009feaff-metrics-certs\") pod \"d3162c67-d852-4c92-8d21-6f85009feaff\" (UID: \"d3162c67-d852-4c92-8d21-6f85009feaff\") " Oct 30 12:34:35.454762 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.454737 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3162c67-d852-4c92-8d21-6f85009feaff-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:35.454969 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.454766 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st82v\" (UniqueName: \"kubernetes.io/projected/d3162c67-d852-4c92-8d21-6f85009feaff-kube-api-access-st82v\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:35.456565 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.456536 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3162c67-d852-4c92-8d21-6f85009feaff-metrics-certs" pod "d3162c67-d852-4c92-8d21-6f85009feaff" (UID: "d3162c67-d852-4c92-8d21-6f85009feaff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:34:35.556164 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:35.556125 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3162c67-d852-4c92-8d21-6f85009feaff-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:36.243120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:36.243050 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx"] Oct 30 12:34:36.328094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:36.323412 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5ce7c4-74c5545cb9-4wqgx"] Oct 30 12:34:37.267158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:37.245616 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:37.268440 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:37.268403 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3162c67-d852-4c92-8d21-6f85009feaff" path="/var/lib/kubelet/pods/d3162c67-d852-4c92-8d21-6f85009feaff/volumes" Oct 30 12:34:37.368103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:37.365166 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:37.384987 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:37.384168 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 12:34:37.384987 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:37.384660 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:37.454227 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:37.454178 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:37.454907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:37.454478 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:37.469607 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:37.454506 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 12:34:37.473278 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:37.473230 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:38.249495 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.247173 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:38.249495 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.247843 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:34:38.410320 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.404773 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:38.410320 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.405232 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 30 12:34:38.410320 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.406475 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:34:38.410320 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.406534 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:34:38.417223 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.417181 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" podUID="f48b6092-fcf0-4787-adef-dba14c5df40f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:38.466097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.417819 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:34:38.466097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.456448 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:38.466097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.457090 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:34:38.545147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.487010 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 12:34:38.545147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.488358 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 12:34:38.545147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.488663 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 12:34:38.545147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.488943 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 12:34:38.545147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.489602 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:34:38.545147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.489657 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:34:38.545147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.489696 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:34:38.836299 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.836203 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 12:34:38.971123 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.970879 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 12:34:38.973606 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.972184 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 12:34:38.973606 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.973189 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 12:34:38.973606 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:38.973240 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 12:34:39.003433 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:39.001962 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 12:34:39.005416 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:39.004536 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z" Oct 30 12:34:39.693174 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:39.689443 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns"] Oct 30 12:34:39.698874 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:39.698835 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp"] Oct 30 12:34:43.657725 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:43.657690 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx"] Oct 30 12:34:43.658312 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:43.657961 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" podUID="3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" containerName="istio-proxy" containerID="cri-o://774d11bb66ccc0cdb68ca19ad81c2a7e1e3cf199eb559e4fbc3be1e9f7d0cb71" gracePeriod=30 Oct 30 12:34:44.689974 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:44.689932 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns" podUID="4b5e5158-cd9b-4722-bd22-b0ab3f3e299d" containerName="payload-processing" containerID="cri-o://8f560a26dd8ec869793d8907acbf57b060d2a34e6bd8488d23d8aad3d1289cb4" gracePeriod=25 Oct 30 12:34:44.740848 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:44.740804 4468 patch_prober.go:29] interesting pod/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:34:44.741003 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:44.740876 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" podUID="3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:34:44.954313 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:44.954233 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns" Oct 30 12:34:44.980714 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:44.980676 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp" podUID="8cd2c069-88b7-4611-b517-c836547c29f2" containerName="payload-pre-processing" containerID="cri-o://dacb57f803f60001fd33bbde4c9c095156b28341c4ee57532a30ea754241a428" gracePeriod=25 Oct 30 12:34:45.030653 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.030619 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b5e5158-cd9b-4722-bd22-b0ab3f3e299d-kube-api-access-84vrp\" (UniqueName: \"kubernetes.io/projected/4b5e5158-cd9b-4722-bd22-b0ab3f3e299d-kube-api-access-84vrp\") pod \"4b5e5158-cd9b-4722-bd22-b0ab3f3e299d\" (UID: \"4b5e5158-cd9b-4722-bd22-b0ab3f3e299d\") " Oct 30 12:34:45.030891 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.030866 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b5e5158-cd9b-4722-bd22-b0ab3f3e299d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b5e5158-cd9b-4722-bd22-b0ab3f3e299d-plugins-config-volume\") pod \"4b5e5158-cd9b-4722-bd22-b0ab3f3e299d\" (UID: \"4b5e5158-cd9b-4722-bd22-b0ab3f3e299d\") " Oct 30 12:34:45.031363 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.031334 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b5e5158-cd9b-4722-bd22-b0ab3f3e299d-plugins-config-volume" pod "4b5e5158-cd9b-4722-bd22-b0ab3f3e299d" (UID: "4b5e5158-cd9b-4722-bd22-b0ab3f3e299d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:45.033425 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.033401 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b5e5158-cd9b-4722-bd22-b0ab3f3e299d-kube-api-access-84vrp" pod "4b5e5158-cd9b-4722-bd22-b0ab3f3e299d" (UID: "4b5e5158-cd9b-4722-bd22-b0ab3f3e299d"). InnerVolumeSpecName "kube-api-access-84vrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:45.092884 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.092847 4468 generic.go:356] "Generic (PLEG): container finished" podID="4b5e5158-cd9b-4722-bd22-b0ab3f3e299d" containerID="8f560a26dd8ec869793d8907acbf57b060d2a34e6bd8488d23d8aad3d1289cb4" exitCode=0 Oct 30 12:34:45.093091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.092913 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns" Oct 30 12:34:45.093091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.092932 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns" event={"ID":"4b5e5158-cd9b-4722-bd22-b0ab3f3e299d","Type":"ContainerDied","Data":"8f560a26dd8ec869793d8907acbf57b060d2a34e6bd8488d23d8aad3d1289cb4"} Oct 30 12:34:45.093091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.093000 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns" event={"ID":"4b5e5158-cd9b-4722-bd22-b0ab3f3e299d","Type":"ContainerDied","Data":"159ee1ae559093d792c187f0d89683f8abfc80e7dd4f103c0636200aacc0e28d"} Oct 30 12:34:45.093091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.093032 4468 scope.go:122] "RemoveContainer" containerID="8f560a26dd8ec869793d8907acbf57b060d2a34e6bd8488d23d8aad3d1289cb4" Oct 30 12:34:45.102716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.102693 4468 scope.go:122] "RemoveContainer" containerID="8f560a26dd8ec869793d8907acbf57b060d2a34e6bd8488d23d8aad3d1289cb4" Oct 30 12:34:45.103026 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:34:45.102994 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f560a26dd8ec869793d8907acbf57b060d2a34e6bd8488d23d8aad3d1289cb4\": container with ID starting with 8f560a26dd8ec869793d8907acbf57b060d2a34e6bd8488d23d8aad3d1289cb4 not found: ID does not exist" containerID="8f560a26dd8ec869793d8907acbf57b060d2a34e6bd8488d23d8aad3d1289cb4" Oct 30 12:34:45.103140 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.103036 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f560a26dd8ec869793d8907acbf57b060d2a34e6bd8488d23d8aad3d1289cb4"} err="failed to get container status \"8f560a26dd8ec869793d8907acbf57b060d2a34e6bd8488d23d8aad3d1289cb4\": rpc error: code = NotFound desc = could not find container \"8f560a26dd8ec869793d8907acbf57b060d2a34e6bd8488d23d8aad3d1289cb4\": container with ID starting with 8f560a26dd8ec869793d8907acbf57b060d2a34e6bd8488d23d8aad3d1289cb4 not found: ID does not exist" Oct 30 12:34:45.132928 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.132889 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84vrp\" (UniqueName: \"kubernetes.io/projected/4b5e5158-cd9b-4722-bd22-b0ab3f3e299d-kube-api-access-84vrp\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:45.132928 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.132928 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b5e5158-cd9b-4722-bd22-b0ab3f3e299d-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:45.155325 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.152808 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns"] Oct 30 12:34:45.159853 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.159823 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cc08f1-575cb5cc76-wf7ns"] Oct 30 12:34:45.293774 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.293748 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp" Oct 30 12:34:45.461166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.461131 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cd2c069-88b7-4611-b517-c836547c29f2-kube-api-access-7r7fj\" (UniqueName: \"kubernetes.io/projected/8cd2c069-88b7-4611-b517-c836547c29f2-kube-api-access-7r7fj\") pod \"8cd2c069-88b7-4611-b517-c836547c29f2\" (UID: \"8cd2c069-88b7-4611-b517-c836547c29f2\") " Oct 30 12:34:45.461326 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.461221 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cd2c069-88b7-4611-b517-c836547c29f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd2c069-88b7-4611-b517-c836547c29f2-plugins-config-volume\") pod \"8cd2c069-88b7-4611-b517-c836547c29f2\" (UID: \"8cd2c069-88b7-4611-b517-c836547c29f2\") " Oct 30 12:34:45.461562 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.461535 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd2c069-88b7-4611-b517-c836547c29f2-plugins-config-volume" pod "8cd2c069-88b7-4611-b517-c836547c29f2" (UID: "8cd2c069-88b7-4611-b517-c836547c29f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:45.461959 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.461929 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd2c069-88b7-4611-b517-c836547c29f2-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:45.463598 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.463573 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd2c069-88b7-4611-b517-c836547c29f2-kube-api-access-7r7fj" pod "8cd2c069-88b7-4611-b517-c836547c29f2" (UID: "8cd2c069-88b7-4611-b517-c836547c29f2"). InnerVolumeSpecName "kube-api-access-7r7fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:45.563061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:45.562973 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r7fj\" (UniqueName: \"kubernetes.io/projected/8cd2c069-88b7-4611-b517-c836547c29f2-kube-api-access-7r7fj\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:46.100456 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:46.100421 4468 generic.go:356] "Generic (PLEG): container finished" podID="8cd2c069-88b7-4611-b517-c836547c29f2" containerID="dacb57f803f60001fd33bbde4c9c095156b28341c4ee57532a30ea754241a428" exitCode=0 Oct 30 12:34:46.100984 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:46.100494 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp" Oct 30 12:34:46.100984 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:46.100508 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp" event={"ID":"8cd2c069-88b7-4611-b517-c836547c29f2","Type":"ContainerDied","Data":"dacb57f803f60001fd33bbde4c9c095156b28341c4ee57532a30ea754241a428"} Oct 30 12:34:46.100984 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:46.100568 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp" event={"ID":"8cd2c069-88b7-4611-b517-c836547c29f2","Type":"ContainerDied","Data":"040ea9f5ff1390788c7b380374f1cb81c574fd800bb44c59e1dda6c91ac66ffd"} Oct 30 12:34:46.100984 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:46.100595 4468 scope.go:122] "RemoveContainer" containerID="dacb57f803f60001fd33bbde4c9c095156b28341c4ee57532a30ea754241a428" Oct 30 12:34:46.111213 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:46.111190 4468 scope.go:122] "RemoveContainer" containerID="dacb57f803f60001fd33bbde4c9c095156b28341c4ee57532a30ea754241a428" Oct 30 12:34:46.111496 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:34:46.111474 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dacb57f803f60001fd33bbde4c9c095156b28341c4ee57532a30ea754241a428\": container with ID starting with dacb57f803f60001fd33bbde4c9c095156b28341c4ee57532a30ea754241a428 not found: ID does not exist" containerID="dacb57f803f60001fd33bbde4c9c095156b28341c4ee57532a30ea754241a428" Oct 30 12:34:46.111555 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:46.111505 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dacb57f803f60001fd33bbde4c9c095156b28341c4ee57532a30ea754241a428"} err="failed to get container status \"dacb57f803f60001fd33bbde4c9c095156b28341c4ee57532a30ea754241a428\": rpc error: code = NotFound desc = could not find container \"dacb57f803f60001fd33bbde4c9c095156b28341c4ee57532a30ea754241a428\": container with ID starting with dacb57f803f60001fd33bbde4c9c095156b28341c4ee57532a30ea754241a428 not found: ID does not exist" Oct 30 12:34:46.165500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:46.165403 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp"] Oct 30 12:34:46.221093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:46.213834 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cc08f1-54c5f4f667-nqtfp"] Oct 30 12:34:47.121093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:47.103568 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b5e5158-cd9b-4722-bd22-b0ab3f3e299d" path="/var/lib/kubelet/pods/4b5e5158-cd9b-4722-bd22-b0ab3f3e299d/volumes" Oct 30 12:34:47.122438 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:47.122404 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cd2c069-88b7-4611-b517-c836547c29f2" path="/var/lib/kubelet/pods/8cd2c069-88b7-4611-b517-c836547c29f2/volumes" Oct 30 12:34:48.949656 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:48.949630 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:34:49.006900 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.006865 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-kube-api-access-mmf5v\" (UniqueName: \"kubernetes.io/projected/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-kube-api-access-mmf5v\") pod \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " Oct 30 12:34:49.007149 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.007002 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-data\") pod \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " Oct 30 12:34:49.007149 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.007036 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-podinfo\") pod \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " Oct 30 12:34:49.007298 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.007182 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-workload-certs\") pod \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " Oct 30 12:34:49.007298 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.007271 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istiod-ca-cert\") pod \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " Oct 30 12:34:49.007409 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.007338 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-envoy\") pod \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " Oct 30 12:34:49.007465 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.007417 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-workload-socket\") pod \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " Oct 30 12:34:49.007521 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.007487 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-token\" (UniqueName: \"kubernetes.io/projected/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-token\") pod \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " Oct 30 12:34:49.007657 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.007637 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-credential-socket\") pod \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\" (UID: \"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4\") " Oct 30 12:34:49.009293 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.009262 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-workload-certs" pod "3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" (UID: "3098f6ce-9631-484f-a7e4-ecdcdffe3fb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:49.009809 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.009784 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-workload-socket" pod "3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" (UID: "3098f6ce-9631-484f-a7e4-ecdcdffe3fb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:49.010198 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.010169 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-credential-socket" pod "3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" (UID: "3098f6ce-9631-484f-a7e4-ecdcdffe3fb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:49.010799 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.010767 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istiod-ca-cert" pod "3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" (UID: "3098f6ce-9631-484f-a7e4-ecdcdffe3fb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:49.011020 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.011002 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-data" pod "3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" (UID: "3098f6ce-9631-484f-a7e4-ecdcdffe3fb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:49.019411 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.019371 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-token" pod "3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" (UID: "3098f6ce-9631-484f-a7e4-ecdcdffe3fb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:49.026206 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.026159 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-kube-api-access-mmf5v" pod "3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" (UID: "3098f6ce-9631-484f-a7e4-ecdcdffe3fb4"). InnerVolumeSpecName "kube-api-access-mmf5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:49.026334 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.026292 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-envoy" pod "3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" (UID: "3098f6ce-9631-484f-a7e4-ecdcdffe3fb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:49.033037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.033011 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-podinfo" pod "3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" (UID: "3098f6ce-9631-484f-a7e4-ecdcdffe3fb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:34:49.109716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.109680 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:49.109716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.109713 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmf5v\" (UniqueName: \"kubernetes.io/projected/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-kube-api-access-mmf5v\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:49.109907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.109727 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:49.109907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.109753 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:49.109907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.109764 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:49.109907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.109776 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:49.109907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.109787 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:49.109907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.109797 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:49.109907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.109808 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:49.222152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.221876 4468 generic.go:356] "Generic (PLEG): container finished" podID="3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" containerID="774d11bb66ccc0cdb68ca19ad81c2a7e1e3cf199eb559e4fbc3be1e9f7d0cb71" exitCode=0 Oct 30 12:34:49.222152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.221992 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" event={"ID":"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4","Type":"ContainerDied","Data":"774d11bb66ccc0cdb68ca19ad81c2a7e1e3cf199eb559e4fbc3be1e9f7d0cb71"} Oct 30 12:34:49.222152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.222048 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" event={"ID":"3098f6ce-9631-484f-a7e4-ecdcdffe3fb4","Type":"ContainerDied","Data":"88f000d9685168444a2f6fd8b2debbc90b75cc1b322c1a713704321aa07e821c"} Oct 30 12:34:49.222152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.222100 4468 scope.go:122] "RemoveContainer" containerID="774d11bb66ccc0cdb68ca19ad81c2a7e1e3cf199eb559e4fbc3be1e9f7d0cb71" Oct 30 12:34:49.222508 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.222233 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx" Oct 30 12:34:49.242406 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.242368 4468 scope.go:122] "RemoveContainer" containerID="9945fe7249d029828d5e8c11ddd1b29b6106cd08c4e3e88318fdf5d6dfa59ca1" Oct 30 12:34:49.252736 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.252704 4468 scope.go:122] "RemoveContainer" containerID="774d11bb66ccc0cdb68ca19ad81c2a7e1e3cf199eb559e4fbc3be1e9f7d0cb71" Oct 30 12:34:49.253062 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:34:49.253039 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"774d11bb66ccc0cdb68ca19ad81c2a7e1e3cf199eb559e4fbc3be1e9f7d0cb71\": container with ID starting with 774d11bb66ccc0cdb68ca19ad81c2a7e1e3cf199eb559e4fbc3be1e9f7d0cb71 not found: ID does not exist" containerID="774d11bb66ccc0cdb68ca19ad81c2a7e1e3cf199eb559e4fbc3be1e9f7d0cb71" Oct 30 12:34:49.253062 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.253088 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"774d11bb66ccc0cdb68ca19ad81c2a7e1e3cf199eb559e4fbc3be1e9f7d0cb71"} err="failed to get container status \"774d11bb66ccc0cdb68ca19ad81c2a7e1e3cf199eb559e4fbc3be1e9f7d0cb71\": rpc error: code = NotFound desc = could not find container \"774d11bb66ccc0cdb68ca19ad81c2a7e1e3cf199eb559e4fbc3be1e9f7d0cb71\": container with ID starting with 774d11bb66ccc0cdb68ca19ad81c2a7e1e3cf199eb559e4fbc3be1e9f7d0cb71 not found: ID does not exist" Oct 30 12:34:49.253234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.253114 4468 scope.go:122] "RemoveContainer" containerID="9945fe7249d029828d5e8c11ddd1b29b6106cd08c4e3e88318fdf5d6dfa59ca1" Oct 30 12:34:49.253393 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:34:49.253375 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9945fe7249d029828d5e8c11ddd1b29b6106cd08c4e3e88318fdf5d6dfa59ca1\": container with ID starting with 9945fe7249d029828d5e8c11ddd1b29b6106cd08c4e3e88318fdf5d6dfa59ca1 not found: ID does not exist" containerID="9945fe7249d029828d5e8c11ddd1b29b6106cd08c4e3e88318fdf5d6dfa59ca1" Oct 30 12:34:49.253430 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.253399 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9945fe7249d029828d5e8c11ddd1b29b6106cd08c4e3e88318fdf5d6dfa59ca1"} err="failed to get container status \"9945fe7249d029828d5e8c11ddd1b29b6106cd08c4e3e88318fdf5d6dfa59ca1\": rpc error: code = NotFound desc = could not find container \"9945fe7249d029828d5e8c11ddd1b29b6106cd08c4e3e88318fdf5d6dfa59ca1\": container with ID starting with 9945fe7249d029828d5e8c11ddd1b29b6106cd08c4e3e88318fdf5d6dfa59ca1 not found: ID does not exist" Oct 30 12:34:49.282391 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.282360 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx"] Oct 30 12:34:49.287892 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:49.287866 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-5ce7c4-openshift-default-67d98f54c7-vkkwx"] Oct 30 12:34:51.090609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.090567 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" path="/var/lib/kubelet/pods/3098f6ce-9631-484f-a7e4-ecdcdffe3fb4/volumes" Oct 30 12:34:51.735406 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.735227 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s"] Oct 30 12:34:51.736101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736063 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" containerName="istio-proxy" Oct 30 12:34:51.736250 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736230 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" containerName="istio-proxy" Oct 30 12:34:51.736375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736258 4468 cpu_manager.go:414] "containerMap: removing container" podUID="aa3ba54c-2c7d-4274-97a4-f6335649d9e7" containerName="payload-pre-processing" Oct 30 12:34:51.736375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736268 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa3ba54c-2c7d-4274-97a4-f6335649d9e7" containerName="payload-pre-processing" Oct 30 12:34:51.736375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736289 4468 cpu_manager.go:414] "containerMap: removing container" podUID="efb3943e-edd9-4e48-affc-17631ef96950" containerName="payload-pre-processing" Oct 30 12:34:51.736375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736297 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb3943e-edd9-4e48-affc-17631ef96950" containerName="payload-pre-processing" Oct 30 12:34:51.736375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736314 4468 cpu_manager.go:414] "containerMap: removing container" podUID="d3162c67-d852-4c92-8d21-6f85009feaff" containerName="maas-api" Oct 30 12:34:51.736375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736321 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3162c67-d852-4c92-8d21-6f85009feaff" containerName="maas-api" Oct 30 12:34:51.736375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736330 4468 cpu_manager.go:414] "containerMap: removing container" podUID="8cd2c069-88b7-4611-b517-c836547c29f2" containerName="payload-pre-processing" Oct 30 12:34:51.736375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736338 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cd2c069-88b7-4611-b517-c836547c29f2" containerName="payload-pre-processing" Oct 30 12:34:51.736375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736350 4468 cpu_manager.go:414] "containerMap: removing container" podUID="43df4a6b-2b51-4575-aff8-530380e15466" containerName="istio-proxy" Oct 30 12:34:51.736375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736357 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43df4a6b-2b51-4575-aff8-530380e15466" containerName="istio-proxy" Oct 30 12:34:51.736375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736379 4468 cpu_manager.go:414] "containerMap: removing container" podUID="81958156-cfb9-4398-a8a3-ca72db1036df" containerName="payload-processing" Oct 30 12:34:51.737041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736387 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81958156-cfb9-4398-a8a3-ca72db1036df" containerName="payload-processing" Oct 30 12:34:51.737041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736432 4468 cpu_manager.go:414] "containerMap: removing container" podUID="4b5e5158-cd9b-4722-bd22-b0ab3f3e299d" containerName="payload-processing" Oct 30 12:34:51.737041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736439 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b5e5158-cd9b-4722-bd22-b0ab3f3e299d" containerName="payload-processing" Oct 30 12:34:51.737041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736756 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3162c67-d852-4c92-8d21-6f85009feaff" containerName="maas-api" Oct 30 12:34:51.737041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736778 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cd2c069-88b7-4611-b517-c836547c29f2" containerName="payload-pre-processing" Oct 30 12:34:51.737041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736788 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="43df4a6b-2b51-4575-aff8-530380e15466" containerName="istio-proxy" Oct 30 12:34:51.737041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736800 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="81958156-cfb9-4398-a8a3-ca72db1036df" containerName="payload-processing" Oct 30 12:34:51.737041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736825 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b5e5158-cd9b-4722-bd22-b0ab3f3e299d" containerName="payload-processing" Oct 30 12:34:51.737041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736835 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" containerName="istio-proxy" Oct 30 12:34:51.737041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736846 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" containerName="istio-proxy" Oct 30 12:34:51.737041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736856 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa3ba54c-2c7d-4274-97a4-f6335649d9e7" containerName="payload-pre-processing" Oct 30 12:34:51.737041 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.736874 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb3943e-edd9-4e48-affc-17631ef96950" containerName="payload-pre-processing" Oct 30 12:34:51.741944 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.741920 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.765408 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.765380 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s"] Oct 30 12:34:51.774299 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.770752 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-327e31-openshift-default-dockercfg-m4bnx\"" Oct 30 12:34:51.782591 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.782561 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4027e387-9043-4303-a6e1-c18365dcfac0-istio-podinfo\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.782753 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.782674 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-istio-envoy\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.782753 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.782726 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-workload-certs\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.782865 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.782783 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-workload-socket\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.782865 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.782816 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-credential-socket\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.782975 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.782920 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-istio-data\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.783030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.783014 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4027e387-9043-4303-a6e1-c18365dcfac0-istio-token\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.783139 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.783119 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nqtd\" (UniqueName: \"kubernetes.io/projected/4027e387-9043-4303-a6e1-c18365dcfac0-kube-api-access-4nqtd\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.783266 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.783247 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4027e387-9043-4303-a6e1-c18365dcfac0-istiod-ca-cert\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.884026 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.883991 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4027e387-9043-4303-a6e1-c18365dcfac0-istiod-ca-cert\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.884228 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.884057 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4027e387-9043-4303-a6e1-c18365dcfac0-istio-podinfo\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.884228 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.884144 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-istio-envoy\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.884228 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.884187 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-workload-certs\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.884228 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.884204 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-workload-socket\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.884228 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.884224 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-credential-socket\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.884531 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.884266 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-istio-data\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.884531 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.884343 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4027e387-9043-4303-a6e1-c18365dcfac0-istio-token\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.884531 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.884403 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nqtd\" (UniqueName: \"kubernetes.io/projected/4027e387-9043-4303-a6e1-c18365dcfac0-kube-api-access-4nqtd\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.884709 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.884617 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-workload-certs\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.884840 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.884816 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-workload-socket\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.884840 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.884827 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4027e387-9043-4303-a6e1-c18365dcfac0-istiod-ca-cert\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.884983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.884917 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-istio-data\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.885023 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.884985 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-credential-socket\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.887394 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.887366 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-istio-envoy\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.887504 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.887459 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4027e387-9043-4303-a6e1-c18365dcfac0-istio-podinfo\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.921748 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.921719 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4027e387-9043-4303-a6e1-c18365dcfac0-istio-token\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:51.921926 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:51.921902 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nqtd\" (UniqueName: \"kubernetes.io/projected/4027e387-9043-4303-a6e1-c18365dcfac0-kube-api-access-4nqtd\") pod \"e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:52.076452 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:52.076362 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:52.458871 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:34:52.458831 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4027e387_9043_4303_a6e1_c18365dcfac0.slice/crio-e108e4a735b3143a5a3fb9fa29add0ef0a87a62a26b1429bddea9577ee25e41f WatchSource:0}: Error finding container e108e4a735b3143a5a3fb9fa29add0ef0a87a62a26b1429bddea9577ee25e41f: Status 404 returned error can't find the container with id e108e4a735b3143a5a3fb9fa29add0ef0a87a62a26b1429bddea9577ee25e41f Oct 30 12:34:52.461826 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:52.461799 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s"] Oct 30 12:34:52.462673 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:52.462641 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:34:52.462729 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:52.462708 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:34:52.462765 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:52.462737 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:34:53.250676 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:53.250639 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" event={"ID":"4027e387-9043-4303-a6e1-c18365dcfac0","Type":"ContainerStarted","Data":"7ca63977ebf67617613e536e0fade02abaa1701a88ea870a25bc7000feb0d398"} Oct 30 12:34:53.250885 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:53.250691 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" event={"ID":"4027e387-9043-4303-a6e1-c18365dcfac0","Type":"ContainerStarted","Data":"e108e4a735b3143a5a3fb9fa29add0ef0a87a62a26b1429bddea9577ee25e41f"} Oct 30 12:34:53.301563 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:53.301513 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" podStartSLOduration=2.301500162 podStartE2EDuration="2.301500162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:53.297515108 +0000 UTC m=+2444.331653503" watchObservedRunningTime="2026-10-30 12:34:53.301500162 +0000 UTC m=+2444.335638558" Oct 30 12:34:53.598760 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:34:53.598675 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:34:54.077205 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:54.077162 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:54.083367 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:54.083336 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:54.257220 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:54.257191 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:54.258141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:54.258064 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:34:56.090031 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.089997 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr"] Oct 30 12:34:56.090798 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.090777 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" containerName="istio-proxy" Oct 30 12:34:56.090798 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.090798 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3098f6ce-9631-484f-a7e4-ecdcdffe3fb4" containerName="istio-proxy" Oct 30 12:34:56.094375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.094354 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" Oct 30 12:34:56.123385 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.121036 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr"] Oct 30 12:34:56.129466 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.126272 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-0bda67-serving-cert\"" Oct 30 12:34:56.165680 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.165638 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh47k\" (UniqueName: \"kubernetes.io/projected/c61a610b-08a7-46ce-b0d5-b2171147362c-kube-api-access-nh47k\") pod \"maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr\" (UID: \"c61a610b-08a7-46ce-b0d5-b2171147362c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" Oct 30 12:34:56.165822 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.165695 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c61a610b-08a7-46ce-b0d5-b2171147362c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr\" (UID: \"c61a610b-08a7-46ce-b0d5-b2171147362c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" Oct 30 12:34:56.165890 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.165842 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c61a610b-08a7-46ce-b0d5-b2171147362c-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr\" (UID: \"c61a610b-08a7-46ce-b0d5-b2171147362c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" Oct 30 12:34:56.165962 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.165941 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61a610b-08a7-46ce-b0d5-b2171147362c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr\" (UID: \"c61a610b-08a7-46ce-b0d5-b2171147362c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" Oct 30 12:34:56.267348 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.267312 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61a610b-08a7-46ce-b0d5-b2171147362c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr\" (UID: \"c61a610b-08a7-46ce-b0d5-b2171147362c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" Oct 30 12:34:56.267588 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.267571 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh47k\" (UniqueName: \"kubernetes.io/projected/c61a610b-08a7-46ce-b0d5-b2171147362c-kube-api-access-nh47k\") pod \"maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr\" (UID: \"c61a610b-08a7-46ce-b0d5-b2171147362c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" Oct 30 12:34:56.267645 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.267615 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c61a610b-08a7-46ce-b0d5-b2171147362c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr\" (UID: \"c61a610b-08a7-46ce-b0d5-b2171147362c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" Oct 30 12:34:56.267718 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.267705 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c61a610b-08a7-46ce-b0d5-b2171147362c-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr\" (UID: \"c61a610b-08a7-46ce-b0d5-b2171147362c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" Oct 30 12:34:56.268051 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.268030 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61a610b-08a7-46ce-b0d5-b2171147362c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr\" (UID: \"c61a610b-08a7-46ce-b0d5-b2171147362c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" Oct 30 12:34:56.270324 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.270299 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c61a610b-08a7-46ce-b0d5-b2171147362c-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr\" (UID: \"c61a610b-08a7-46ce-b0d5-b2171147362c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" Oct 30 12:34:56.270448 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.270432 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c61a610b-08a7-46ce-b0d5-b2171147362c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr\" (UID: \"c61a610b-08a7-46ce-b0d5-b2171147362c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" Oct 30 12:34:56.316091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.315526 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh47k\" (UniqueName: \"kubernetes.io/projected/c61a610b-08a7-46ce-b0d5-b2171147362c-kube-api-access-nh47k\") pod \"maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr\" (UID: \"c61a610b-08a7-46ce-b0d5-b2171147362c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" Oct 30 12:34:56.411635 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.411600 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" Oct 30 12:34:56.717436 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:34:56.717398 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61a610b_08a7_46ce_b0d5_b2171147362c.slice/crio-2510d1b9ab9a7d890b0590e23b9b000fe28deaab23a596253b10b8fa91a80370 WatchSource:0}: Error finding container 2510d1b9ab9a7d890b0590e23b9b000fe28deaab23a596253b10b8fa91a80370: Status 404 returned error can't find the container with id 2510d1b9ab9a7d890b0590e23b9b000fe28deaab23a596253b10b8fa91a80370 Oct 30 12:34:56.720687 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:56.720665 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr"] Oct 30 12:34:57.277899 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.277856 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" event={"ID":"c61a610b-08a7-46ce-b0d5-b2171147362c","Type":"ContainerStarted","Data":"2a77598c49ac829099e09c80072868d9536c8623a5bc20d7dbdf44f5a8dd7e58"} Oct 30 12:34:57.278306 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.277928 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" event={"ID":"c61a610b-08a7-46ce-b0d5-b2171147362c","Type":"ContainerStarted","Data":"2510d1b9ab9a7d890b0590e23b9b000fe28deaab23a596253b10b8fa91a80370"} Oct 30 12:34:57.279024 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.279001 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" Oct 30 12:34:57.328611 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.328563 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" podStartSLOduration=0.936718109 podStartE2EDuration="1.328550452s" totalImagesPullingTime="391.832343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:57.327444085 +0000 UTC m=+2448.361582481" watchObservedRunningTime="2026-10-30 12:34:57.328550452 +0000 UTC m=+2448.362688847" Oct 30 12:34:57.750813 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.750775 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67"] Oct 30 12:34:57.754815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.754795 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.774446 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.774424 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-47fd99-openshift-default-dockercfg-lmqzd\"" Oct 30 12:34:57.784175 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.784150 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67"] Oct 30 12:34:57.811793 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.811760 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.811963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.811809 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmc5h\" (UniqueName: \"kubernetes.io/projected/4fc449d1-bb52-4529-bb7c-407d5277050f-kube-api-access-rmc5h\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.811963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.811924 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.812082 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.812041 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.812167 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.812148 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.812238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.812220 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.812319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.812299 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc449d1-bb52-4529-bb7c-407d5277050f-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.812356 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.812347 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.812480 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.812466 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.914096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.914038 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.914262 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.914105 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmc5h\" (UniqueName: \"kubernetes.io/projected/4fc449d1-bb52-4529-bb7c-407d5277050f-kube-api-access-rmc5h\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.914262 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.914147 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.914262 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.914217 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.914379 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.914285 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.914379 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.914312 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.914379 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.914353 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc449d1-bb52-4529-bb7c-407d5277050f-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.914478 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.914394 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.914513 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.914494 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.914598 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.914571 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.914776 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.914757 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.914823 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.914496 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.915024 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.914994 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.915190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.915134 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc449d1-bb52-4529-bb7c-407d5277050f-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.917297 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.917271 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.917602 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.917578 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.940973 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.940934 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:57.941350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:57.941328 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmc5h\" (UniqueName: \"kubernetes.io/projected/4fc449d1-bb52-4529-bb7c-407d5277050f-kube-api-access-rmc5h\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:58.087154 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.070319 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:34:58.184108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.139298 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn"] Oct 30 12:34:58.184108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.139686 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" podUID="dca98735-7b6b-41bc-b651-b21358aab369" containerName="maas-api" containerID="cri-o://57c71c5a6275d22b1935a177f4e2755c6176cb9d267d2450c046a0f2bcabc685" gracePeriod=30 Oct 30 12:34:58.593034 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.593009 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" Oct 30 12:34:58.606683 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:34:58.606652 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc449d1_bb52_4529_bb7c_407d5277050f.slice/crio-a246717f7ade0b11b73999be6eefd701b75814b7d3e62130f8995a70a1a81cfa WatchSource:0}: Error finding container a246717f7ade0b11b73999be6eefd701b75814b7d3e62130f8995a70a1a81cfa: Status 404 returned error can't find the container with id a246717f7ade0b11b73999be6eefd701b75814b7d3e62130f8995a70a1a81cfa Oct 30 12:34:58.613189 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.613158 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67"] Oct 30 12:34:58.614889 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.614848 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:34:58.615010 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.614933 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:34:58.615010 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.614977 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:34:58.848882 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.848850 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dca98735-7b6b-41bc-b651-b21358aab369-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca98735-7b6b-41bc-b651-b21358aab369-maas-api-tls\") pod \"dca98735-7b6b-41bc-b651-b21358aab369\" (UID: \"dca98735-7b6b-41bc-b651-b21358aab369\") " Oct 30 12:34:58.849035 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.848998 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca98735-7b6b-41bc-b651-b21358aab369-kube-api-access-b4xts\" (UniqueName: \"kubernetes.io/projected/dca98735-7b6b-41bc-b651-b21358aab369-kube-api-access-b4xts\") pod \"dca98735-7b6b-41bc-b651-b21358aab369\" (UID: \"dca98735-7b6b-41bc-b651-b21358aab369\") " Oct 30 12:34:58.849146 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.849121 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dca98735-7b6b-41bc-b651-b21358aab369-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca98735-7b6b-41bc-b651-b21358aab369-metrics-certs\") pod \"dca98735-7b6b-41bc-b651-b21358aab369\" (UID: \"dca98735-7b6b-41bc-b651-b21358aab369\") " Oct 30 12:34:58.849241 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.849220 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dca98735-7b6b-41bc-b651-b21358aab369-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca98735-7b6b-41bc-b651-b21358aab369-openshift-service-ca\") pod \"dca98735-7b6b-41bc-b651-b21358aab369\" (UID: \"dca98735-7b6b-41bc-b651-b21358aab369\") " Oct 30 12:34:58.849962 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.849941 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca98735-7b6b-41bc-b651-b21358aab369-openshift-service-ca" pod "dca98735-7b6b-41bc-b651-b21358aab369" (UID: "dca98735-7b6b-41bc-b651-b21358aab369"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:58.851193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.851172 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca98735-7b6b-41bc-b651-b21358aab369-kube-api-access-b4xts" pod "dca98735-7b6b-41bc-b651-b21358aab369" (UID: "dca98735-7b6b-41bc-b651-b21358aab369"). InnerVolumeSpecName "kube-api-access-b4xts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:58.851596 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.851575 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dca98735-7b6b-41bc-b651-b21358aab369-metrics-certs" pod "dca98735-7b6b-41bc-b651-b21358aab369" (UID: "dca98735-7b6b-41bc-b651-b21358aab369"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:34:58.851682 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.851606 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dca98735-7b6b-41bc-b651-b21358aab369-maas-api-tls" pod "dca98735-7b6b-41bc-b651-b21358aab369" (UID: "dca98735-7b6b-41bc-b651-b21358aab369"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:34:58.951425 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.951349 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca98735-7b6b-41bc-b651-b21358aab369-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:58.951425 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.951377 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4xts\" (UniqueName: \"kubernetes.io/projected/dca98735-7b6b-41bc-b651-b21358aab369-kube-api-access-b4xts\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:58.951425 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.951387 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca98735-7b6b-41bc-b651-b21358aab369-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:58.951425 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:58.951395 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca98735-7b6b-41bc-b651-b21358aab369-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:34:59.294958 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:59.294924 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" event={"ID":"4fc449d1-bb52-4529-bb7c-407d5277050f","Type":"ContainerStarted","Data":"65ca75bfbcb6d215d07e515da0bb67faf8a5184f9841d68372e3fd5f1cc26d00"} Oct 30 12:34:59.295152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:59.294979 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" event={"ID":"4fc449d1-bb52-4529-bb7c-407d5277050f","Type":"ContainerStarted","Data":"a246717f7ade0b11b73999be6eefd701b75814b7d3e62130f8995a70a1a81cfa"} Oct 30 12:34:59.296292 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:59.296267 4468 generic.go:356] "Generic (PLEG): container finished" podID="dca98735-7b6b-41bc-b651-b21358aab369" containerID="57c71c5a6275d22b1935a177f4e2755c6176cb9d267d2450c046a0f2bcabc685" exitCode=0 Oct 30 12:34:59.296390 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:59.296304 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" Oct 30 12:34:59.296390 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:59.296350 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" event={"ID":"dca98735-7b6b-41bc-b651-b21358aab369","Type":"ContainerDied","Data":"57c71c5a6275d22b1935a177f4e2755c6176cb9d267d2450c046a0f2bcabc685"} Oct 30 12:34:59.296454 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:59.296400 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn" event={"ID":"dca98735-7b6b-41bc-b651-b21358aab369","Type":"ContainerDied","Data":"a94e50b9d33b62da17a2e9d0488f703e20af66f9face4292dfcf106b712ccc12"} Oct 30 12:34:59.296454 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:59.296425 4468 scope.go:122] "RemoveContainer" containerID="57c71c5a6275d22b1935a177f4e2755c6176cb9d267d2450c046a0f2bcabc685" Oct 30 12:34:59.307623 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:59.307601 4468 scope.go:122] "RemoveContainer" containerID="57c71c5a6275d22b1935a177f4e2755c6176cb9d267d2450c046a0f2bcabc685" Oct 30 12:34:59.308221 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:34:59.308197 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c71c5a6275d22b1935a177f4e2755c6176cb9d267d2450c046a0f2bcabc685\": container with ID starting with 57c71c5a6275d22b1935a177f4e2755c6176cb9d267d2450c046a0f2bcabc685 not found: ID does not exist" containerID="57c71c5a6275d22b1935a177f4e2755c6176cb9d267d2450c046a0f2bcabc685" Oct 30 12:34:59.308306 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:59.308287 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c71c5a6275d22b1935a177f4e2755c6176cb9d267d2450c046a0f2bcabc685"} err="failed to get container status \"57c71c5a6275d22b1935a177f4e2755c6176cb9d267d2450c046a0f2bcabc685\": rpc error: code = NotFound desc = could not find container \"57c71c5a6275d22b1935a177f4e2755c6176cb9d267d2450c046a0f2bcabc685\": container with ID starting with 57c71c5a6275d22b1935a177f4e2755c6176cb9d267d2450c046a0f2bcabc685 not found: ID does not exist" Oct 30 12:34:59.344293 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:59.344244 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" podStartSLOduration=2.344230921 podStartE2EDuration="2.344230921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:59.342907577 +0000 UTC m=+2450.377045975" watchObservedRunningTime="2026-10-30 12:34:59.344230921 +0000 UTC m=+2450.378369316" Oct 30 12:34:59.385952 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:59.385922 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn"] Oct 30 12:34:59.390029 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:34:59.389990 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc08f1-75c47bc8c7-f7vzn"] Oct 30 12:35:00.074763 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:00.074731 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:35:00.076232 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:00.076209 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 12:35:00.076298 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:00.076256 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" podUID="4fc449d1-bb52-4529-bb7c-407d5277050f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 12:35:01.074928 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:01.074891 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 12:35:01.075360 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:01.074952 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" podUID="4fc449d1-bb52-4529-bb7c-407d5277050f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 12:35:01.090245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:01.090209 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dca98735-7b6b-41bc-b651-b21358aab369" path="/var/lib/kubelet/pods/dca98735-7b6b-41bc-b651-b21358aab369/volumes" Oct 30 12:35:02.074737 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:02.074705 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 12:35:02.074914 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:02.074760 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" podUID="4fc449d1-bb52-4529-bb7c-407d5277050f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 12:35:03.075375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:03.075338 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 12:35:03.075860 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:03.075406 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" podUID="4fc449d1-bb52-4529-bb7c-407d5277050f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 12:35:04.075532 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:04.075493 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 12:35:04.075926 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:04.075555 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" podUID="4fc449d1-bb52-4529-bb7c-407d5277050f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 12:35:04.301885 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:04.301857 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" Oct 30 12:35:05.074558 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:05.074525 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 12:35:05.074710 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:05.074580 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" podUID="4fc449d1-bb52-4529-bb7c-407d5277050f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 12:35:06.322919 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:06.322884 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:35:06.323379 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:06.323359 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:35:06.324459 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:06.324441 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:35:06.569902 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:06.569872 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x"] Oct 30 12:35:06.570140 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:06.570083 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" podUID="f9f83c6f-04b8-46a1-824a-da232fe7c23d" containerName="istio-proxy" containerID="cri-o://ae493a849ba07886a3112b65006e7d4427a8b80df1f2ec259768476b8cc489a2" gracePeriod=30 Oct 30 12:35:10.269932 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.269895 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd"] Oct 30 12:35:10.270793 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.270774 4468 cpu_manager.go:414] "containerMap: removing container" podUID="dca98735-7b6b-41bc-b651-b21358aab369" containerName="maas-api" Oct 30 12:35:10.270793 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.270794 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dca98735-7b6b-41bc-b651-b21358aab369" containerName="maas-api" Oct 30 12:35:10.271000 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.270969 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="dca98735-7b6b-41bc-b651-b21358aab369" containerName="maas-api" Oct 30 12:35:10.274888 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.274869 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.294707 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.294624 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd"] Oct 30 12:35:10.308256 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.308228 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-9d45fb-openshift-default-dockercfg-xd9x9\"" Oct 30 12:35:10.410539 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.410500 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.410743 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.410632 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.410743 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.410664 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.410880 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.410771 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e2eab8-2195-41e0-97a7-63383ccbbe48-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.410880 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.410806 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.410995 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.410885 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.410995 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.410955 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.411157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.410991 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h8v6\" (UniqueName: \"kubernetes.io/projected/73e2eab8-2195-41e0-97a7-63383ccbbe48-kube-api-access-4h8v6\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.411157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.411024 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.533350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.533255 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e2eab8-2195-41e0-97a7-63383ccbbe48-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.533350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.533305 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.533629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.533354 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.533629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.533413 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.533629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.533451 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h8v6\" (UniqueName: \"kubernetes.io/projected/73e2eab8-2195-41e0-97a7-63383ccbbe48-kube-api-access-4h8v6\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.533629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.533483 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.533832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.533699 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.533832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.533784 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.533832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.533811 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.535215 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.535187 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.536908 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.535561 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.537038 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.536117 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e2eab8-2195-41e0-97a7-63383ccbbe48-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.537204 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.536314 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.537305 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.536501 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.568371 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.568333 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.569124 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.568659 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.571205 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.570937 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.572007 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.571959 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h8v6\" (UniqueName: \"kubernetes.io/projected/73e2eab8-2195-41e0-97a7-63383ccbbe48-kube-api-access-4h8v6\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.604234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.603990 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:10.948485 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:35:10.948432 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e2eab8_2195_41e0_97a7_63383ccbbe48.slice/crio-7953dfb24fea2c27bfffeb03175cb1054a6bc499bbde997232a5da4f13f7d043 WatchSource:0}: Error finding container 7953dfb24fea2c27bfffeb03175cb1054a6bc499bbde997232a5da4f13f7d043: Status 404 returned error can't find the container with id 7953dfb24fea2c27bfffeb03175cb1054a6bc499bbde997232a5da4f13f7d043 Oct 30 12:35:10.951818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.951366 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:35:10.951818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.951461 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:35:10.951818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.951503 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:35:10.952025 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:10.951884 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd"] Oct 30 12:35:11.494994 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:11.494953 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" event={"ID":"73e2eab8-2195-41e0-97a7-63383ccbbe48","Type":"ContainerStarted","Data":"454ccb1c17bbc3e0b94fc856c34c9b6f5ce6efcd14552010891686d4a0fc55b2"} Oct 30 12:35:11.495473 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:11.495018 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" event={"ID":"73e2eab8-2195-41e0-97a7-63383ccbbe48","Type":"ContainerStarted","Data":"7953dfb24fea2c27bfffeb03175cb1054a6bc499bbde997232a5da4f13f7d043"} Oct 30 12:35:11.560775 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:11.560725 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" podStartSLOduration=1.5607107839999999 podStartE2EDuration="1.560710784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:11.555986311 +0000 UTC m=+2462.590124707" watchObservedRunningTime="2026-10-30 12:35:11.560710784 +0000 UTC m=+2462.594849179" Oct 30 12:35:11.605370 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:11.605342 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:11.883415 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:11.883388 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:35:12.057217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.057121 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-credential-socket\") pod \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " Oct 30 12:35:12.057217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.057212 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-workload-certs\") pod \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " Oct 30 12:35:12.057447 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.057287 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-token\" (UniqueName: \"kubernetes.io/projected/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-token\") pod \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " Oct 30 12:35:12.057447 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.057359 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9f83c6f-04b8-46a1-824a-da232fe7c23d-kube-api-access-s767l\" (UniqueName: \"kubernetes.io/projected/f9f83c6f-04b8-46a1-824a-da232fe7c23d-kube-api-access-s767l\") pod \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " Oct 30 12:35:12.057559 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.057457 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istiod-ca-cert\") pod \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " Oct 30 12:35:12.057559 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.057527 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-workload-socket\") pod \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " Oct 30 12:35:12.057559 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.057531 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-workload-certs" pod "f9f83c6f-04b8-46a1-824a-da232fe7c23d" (UID: "f9f83c6f-04b8-46a1-824a-da232fe7c23d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:35:12.057716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.057575 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-data\") pod \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " Oct 30 12:35:12.057716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.057634 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-podinfo\") pod \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " Oct 30 12:35:12.057716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.057662 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-envoy\") pod \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\" (UID: \"f9f83c6f-04b8-46a1-824a-da232fe7c23d\") " Oct 30 12:35:12.057885 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.057801 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-workload-socket" pod "f9f83c6f-04b8-46a1-824a-da232fe7c23d" (UID: "f9f83c6f-04b8-46a1-824a-da232fe7c23d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:35:12.057885 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.057865 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-data" pod "f9f83c6f-04b8-46a1-824a-da232fe7c23d" (UID: "f9f83c6f-04b8-46a1-824a-da232fe7c23d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:35:12.057984 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.057906 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istiod-ca-cert" pod "f9f83c6f-04b8-46a1-824a-da232fe7c23d" (UID: "f9f83c6f-04b8-46a1-824a-da232fe7c23d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:35:12.058313 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.058291 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:35:12.058478 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.058464 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:35:12.058577 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.058558 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:35:12.058577 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.058578 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:35:12.059506 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.059476 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-credential-socket" pod "f9f83c6f-04b8-46a1-824a-da232fe7c23d" (UID: "f9f83c6f-04b8-46a1-824a-da232fe7c23d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:35:12.060003 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.059975 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-token" pod "f9f83c6f-04b8-46a1-824a-da232fe7c23d" (UID: "f9f83c6f-04b8-46a1-824a-da232fe7c23d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:35:12.060089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.060039 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-podinfo" pod "f9f83c6f-04b8-46a1-824a-da232fe7c23d" (UID: "f9f83c6f-04b8-46a1-824a-da232fe7c23d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:35:12.060382 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.060362 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f83c6f-04b8-46a1-824a-da232fe7c23d-kube-api-access-s767l" pod "f9f83c6f-04b8-46a1-824a-da232fe7c23d" (UID: "f9f83c6f-04b8-46a1-824a-da232fe7c23d"). InnerVolumeSpecName "kube-api-access-s767l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:35:12.060430 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.060403 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-envoy" pod "f9f83c6f-04b8-46a1-824a-da232fe7c23d" (UID: "f9f83c6f-04b8-46a1-824a-da232fe7c23d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:35:12.159811 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.159777 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:35:12.159811 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.159803 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:35:12.159811 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.159814 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s767l\" (UniqueName: \"kubernetes.io/projected/f9f83c6f-04b8-46a1-824a-da232fe7c23d-kube-api-access-s767l\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:35:12.159811 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.159824 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:35:12.160241 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.159835 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9f83c6f-04b8-46a1-824a-da232fe7c23d-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:35:12.502307 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.502273 4468 generic.go:356] "Generic (PLEG): container finished" podID="f9f83c6f-04b8-46a1-824a-da232fe7c23d" containerID="ae493a849ba07886a3112b65006e7d4427a8b80df1f2ec259768476b8cc489a2" exitCode=0 Oct 30 12:35:12.502832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.502331 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" Oct 30 12:35:12.502832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.502356 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" event={"ID":"f9f83c6f-04b8-46a1-824a-da232fe7c23d","Type":"ContainerDied","Data":"ae493a849ba07886a3112b65006e7d4427a8b80df1f2ec259768476b8cc489a2"} Oct 30 12:35:12.502832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.502442 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x" event={"ID":"f9f83c6f-04b8-46a1-824a-da232fe7c23d","Type":"ContainerDied","Data":"26d03b3e633f13de27dc380823444aaef8a5b0e5107d205020c138efb557f6ad"} Oct 30 12:35:12.502832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.502464 4468 scope.go:122] "RemoveContainer" containerID="ae493a849ba07886a3112b65006e7d4427a8b80df1f2ec259768476b8cc489a2" Oct 30 12:35:12.512597 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.512581 4468 scope.go:122] "RemoveContainer" containerID="c0fa5408a45f5b4d0b5d12f2afb6f025f883d5dff83297a912598d0734f468cf" Oct 30 12:35:12.523016 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.522999 4468 scope.go:122] "RemoveContainer" containerID="ae493a849ba07886a3112b65006e7d4427a8b80df1f2ec259768476b8cc489a2" Oct 30 12:35:12.523313 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:35:12.523294 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae493a849ba07886a3112b65006e7d4427a8b80df1f2ec259768476b8cc489a2\": container with ID starting with ae493a849ba07886a3112b65006e7d4427a8b80df1f2ec259768476b8cc489a2 not found: ID does not exist" containerID="ae493a849ba07886a3112b65006e7d4427a8b80df1f2ec259768476b8cc489a2" Oct 30 12:35:12.523361 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.523323 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae493a849ba07886a3112b65006e7d4427a8b80df1f2ec259768476b8cc489a2"} err="failed to get container status \"ae493a849ba07886a3112b65006e7d4427a8b80df1f2ec259768476b8cc489a2\": rpc error: code = NotFound desc = could not find container \"ae493a849ba07886a3112b65006e7d4427a8b80df1f2ec259768476b8cc489a2\": container with ID starting with ae493a849ba07886a3112b65006e7d4427a8b80df1f2ec259768476b8cc489a2 not found: ID does not exist" Oct 30 12:35:12.523361 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.523339 4468 scope.go:122] "RemoveContainer" containerID="c0fa5408a45f5b4d0b5d12f2afb6f025f883d5dff83297a912598d0734f468cf" Oct 30 12:35:12.523587 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:35:12.523570 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0fa5408a45f5b4d0b5d12f2afb6f025f883d5dff83297a912598d0734f468cf\": container with ID starting with c0fa5408a45f5b4d0b5d12f2afb6f025f883d5dff83297a912598d0734f468cf not found: ID does not exist" containerID="c0fa5408a45f5b4d0b5d12f2afb6f025f883d5dff83297a912598d0734f468cf" Oct 30 12:35:12.523626 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.523595 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0fa5408a45f5b4d0b5d12f2afb6f025f883d5dff83297a912598d0734f468cf"} err="failed to get container status \"c0fa5408a45f5b4d0b5d12f2afb6f025f883d5dff83297a912598d0734f468cf\": rpc error: code = NotFound desc = could not find container \"c0fa5408a45f5b4d0b5d12f2afb6f025f883d5dff83297a912598d0734f468cf\": container with ID starting with c0fa5408a45f5b4d0b5d12f2afb6f025f883d5dff83297a912598d0734f468cf not found: ID does not exist" Oct 30 12:35:12.564425 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.564399 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x"] Oct 30 12:35:12.577835 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.577811 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cc08f1-openshift-default-6b6864bbb5-ftx2x"] Oct 30 12:35:12.610655 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.610632 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:12.782896 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.782817 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt"] Oct 30 12:35:12.783416 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.783402 4468 cpu_manager.go:414] "containerMap: removing container" podUID="f9f83c6f-04b8-46a1-824a-da232fe7c23d" containerName="istio-proxy" Oct 30 12:35:12.783416 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.783416 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9f83c6f-04b8-46a1-824a-da232fe7c23d" containerName="istio-proxy" Oct 30 12:35:12.783540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.783434 4468 cpu_manager.go:414] "containerMap: removing container" podUID="f9f83c6f-04b8-46a1-824a-da232fe7c23d" containerName="istio-proxy" Oct 30 12:35:12.783540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.783439 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9f83c6f-04b8-46a1-824a-da232fe7c23d" containerName="istio-proxy" Oct 30 12:35:12.783540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.783522 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9f83c6f-04b8-46a1-824a-da232fe7c23d" containerName="istio-proxy" Oct 30 12:35:12.783540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.783532 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9f83c6f-04b8-46a1-824a-da232fe7c23d" containerName="istio-proxy" Oct 30 12:35:12.788612 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.788591 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:12.808976 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.808952 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt"] Oct 30 12:35:12.978289 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.978249 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:12.978471 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.978381 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:12.978471 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.978429 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtkx6\" (UniqueName: \"kubernetes.io/projected/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-kube-api-access-vtkx6\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:12.978609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.978534 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:12.978609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.978579 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:12.978779 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.978754 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:12.978844 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.978827 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:12.978911 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.978896 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:12.978977 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:12.978941 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.079941 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.079847 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.079941 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.079915 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.079941 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.079943 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtkx6\" (UniqueName: \"kubernetes.io/projected/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-kube-api-access-vtkx6\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.080302 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.079973 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.080302 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.079995 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.080302 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.080048 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.080302 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.080130 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.080302 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.080196 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.080302 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.080240 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.080619 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.080360 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.080619 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.080383 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.080619 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.080504 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.080763 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.080676 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.081114 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.081063 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.082576 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.082552 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.082676 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.082642 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.091450 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.091424 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9f83c6f-04b8-46a1-824a-da232fe7c23d" path="/var/lib/kubelet/pods/f9f83c6f-04b8-46a1-824a-da232fe7c23d/volumes" Oct 30 12:35:13.104754 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.104726 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.104940 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.104924 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtkx6\" (UniqueName: \"kubernetes.io/projected/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-kube-api-access-vtkx6\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.404511 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.404415 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:13.521820 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.521789 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:13.522858 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.522834 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:35:13.724240 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:35:13.724197 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7abde5f_37e2_40d0_9ddc_bd907e115aa0.slice/crio-848334bf89b56fe026b8fca853a15b3ec190cfcb34b071434a15fb0a0b05b981 WatchSource:0}: Error finding container 848334bf89b56fe026b8fca853a15b3ec190cfcb34b071434a15fb0a0b05b981: Status 404 returned error can't find the container with id 848334bf89b56fe026b8fca853a15b3ec190cfcb34b071434a15fb0a0b05b981 Oct 30 12:35:13.727465 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.727419 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:35:13.727559 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.727517 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:35:13.727612 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.727562 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:35:13.727833 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.727813 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt"] Oct 30 12:35:13.778839 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.778813 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p"] Oct 30 12:35:13.784187 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.784116 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" Oct 30 12:35:13.791411 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.791379 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p\" (UID: \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" Oct 30 12:35:13.791575 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.791558 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8pxg\" (UniqueName: \"kubernetes.io/projected/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-kube-api-access-d8pxg\") pod \"maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p\" (UID: \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" Oct 30 12:35:13.791756 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.791726 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p\" (UID: \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" Oct 30 12:35:13.791877 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.791794 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p\" (UID: \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" Oct 30 12:35:13.805185 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.800365 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-327e31-serving-cert\"" Oct 30 12:35:13.806527 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.806503 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p"] Oct 30 12:35:13.892917 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.892878 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8pxg\" (UniqueName: \"kubernetes.io/projected/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-kube-api-access-d8pxg\") pod \"maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p\" (UID: \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" Oct 30 12:35:13.893193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.892992 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p\" (UID: \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" Oct 30 12:35:13.893193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.893042 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p\" (UID: \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" Oct 30 12:35:13.893193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.893150 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p\" (UID: \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" Oct 30 12:35:13.893506 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:35:13.893332 4468 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-serving-cert: secret "maas-api-e2e-shared-a-w4-327e31-serving-cert" not found Oct 30 12:35:13.893506 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:35:13.893391 4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-maas-api-tls podName:4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd nodeName:}" failed. No retries permitted until 2026-10-30 12:35:14.393371479 +0000 UTC m=+2465.427509859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-maas-api-tls") pod "maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" (UID: "4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd") : secret "maas-api-e2e-shared-a-w4-327e31-serving-cert" not found Oct 30 12:35:13.893989 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.893962 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p\" (UID: \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" Oct 30 12:35:13.895996 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.895972 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p\" (UID: \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" Oct 30 12:35:13.918368 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:13.918334 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8pxg\" (UniqueName: \"kubernetes.io/projected/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-kube-api-access-d8pxg\") pod \"maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p\" (UID: \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" Oct 30 12:35:14.400681 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:14.400646 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p\" (UID: \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" Oct 30 12:35:14.403039 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:14.403019 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p\" (UID: \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" Oct 30 12:35:14.551697 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:14.551653 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" event={"ID":"b7abde5f-37e2-40d0-9ddc-bd907e115aa0","Type":"ContainerStarted","Data":"a50e1660fc0db064e39bfea6e4a5c6743762d77870e738cf16ad987d9cf4f198"} Oct 30 12:35:14.552189 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:14.551724 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" event={"ID":"b7abde5f-37e2-40d0-9ddc-bd907e115aa0","Type":"ContainerStarted","Data":"848334bf89b56fe026b8fca853a15b3ec190cfcb34b071434a15fb0a0b05b981"} Oct 30 12:35:14.707093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:14.703375 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" Oct 30 12:35:15.042513 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:15.042436 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" podStartSLOduration=3.042415236 podStartE2EDuration="3.042415236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:14.638096971 +0000 UTC m=+2465.672235367" watchObservedRunningTime="2026-10-30 12:35:15.042415236 +0000 UTC m=+2466.076553633" Oct 30 12:35:15.044658 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:15.044634 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6"] Oct 30 12:35:15.045711 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:35:15.045679 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9fd4ee_b6d4_4c72_9a09_3e0bf849e3cd.slice/crio-0a1fe9ec60f1223cd7e9f1ef6d9a0ee2eb1b237a262f12e79fbc418b4ecf13fd WatchSource:0}: Error finding container 0a1fe9ec60f1223cd7e9f1ef6d9a0ee2eb1b237a262f12e79fbc418b4ecf13fd: Status 404 returned error can't find the container with id 0a1fe9ec60f1223cd7e9f1ef6d9a0ee2eb1b237a262f12e79fbc418b4ecf13fd Oct 30 12:35:15.051592 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:15.051566 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p"] Oct 30 12:35:15.058248 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:15.058223 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-748f7476-dnbw6"] Oct 30 12:35:15.089602 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:15.089568 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="615f8b23-a00e-4273-9b34-0c704075965f" path="/var/lib/kubelet/pods/615f8b23-a00e-4273-9b34-0c704075965f/volumes" Oct 30 12:35:15.414152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:15.408508 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:15.432089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:15.416739 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 12:35:15.432089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:15.416814 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" podUID="b7abde5f-37e2-40d0-9ddc-bd907e115aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 12:35:15.567147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:15.566830 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" event={"ID":"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd","Type":"ContainerStarted","Data":"0a1fe9ec60f1223cd7e9f1ef6d9a0ee2eb1b237a262f12e79fbc418b4ecf13fd"} Oct 30 12:35:16.238165 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:16.238129 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm"] Oct 30 12:35:16.243028 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:16.243004 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" Oct 30 12:35:16.244916 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:16.244886 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm"] Oct 30 12:35:16.258963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:16.258940 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-9d45fb\"" Oct 30 12:35:16.259397 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:16.259377 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-9d45fb-dockercfg-t7qwg\"" Oct 30 12:35:16.418099 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:16.413609 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 12:35:16.418099 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:16.413673 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" podUID="b7abde5f-37e2-40d0-9ddc-bd907e115aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 12:35:16.444097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:16.442532 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ef76d8-4829-4123-bbff-e7b36c59afba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm\" (UID: \"35ef76d8-4829-4123-bbff-e7b36c59afba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" Oct 30 12:35:16.444097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:16.442743 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvdml\" (UniqueName: \"kubernetes.io/projected/35ef76d8-4829-4123-bbff-e7b36c59afba-kube-api-access-rvdml\") pod \"payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm\" (UID: \"35ef76d8-4829-4123-bbff-e7b36c59afba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" Oct 30 12:35:16.554155 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:16.553990 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvdml\" (UniqueName: \"kubernetes.io/projected/35ef76d8-4829-4123-bbff-e7b36c59afba-kube-api-access-rvdml\") pod \"payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm\" (UID: \"35ef76d8-4829-4123-bbff-e7b36c59afba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" Oct 30 12:35:16.580152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:16.566284 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ef76d8-4829-4123-bbff-e7b36c59afba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm\" (UID: \"35ef76d8-4829-4123-bbff-e7b36c59afba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" Oct 30 12:35:16.582154 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:16.582037 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ef76d8-4829-4123-bbff-e7b36c59afba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm\" (UID: \"35ef76d8-4829-4123-bbff-e7b36c59afba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" Oct 30 12:35:16.602721 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:16.602678 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" event={"ID":"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd","Type":"ContainerStarted","Data":"06ab4a4bd6c138d58f54bba8f63c4d20a8942f2e25f2b94bb94c24bb79eb606e"} Oct 30 12:35:16.602923 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:16.602892 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" Oct 30 12:35:16.634897 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:16.628876 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvdml\" (UniqueName: \"kubernetes.io/projected/35ef76d8-4829-4123-bbff-e7b36c59afba-kube-api-access-rvdml\") pod \"payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm\" (UID: \"35ef76d8-4829-4123-bbff-e7b36c59afba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" Oct 30 12:35:16.660306 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:16.660254 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" podStartSLOduration=3.149124528 podStartE2EDuration="3.660239999s" totalImagesPullingTime="511.115471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:16.657898674 +0000 UTC m=+2467.692037070" watchObservedRunningTime="2026-10-30 12:35:16.660239999 +0000 UTC m=+2467.694378394" Oct 30 12:35:16.860736 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:16.860645 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" Oct 30 12:35:17.235616 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:17.235586 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm"] Oct 30 12:35:17.239342 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:35:17.239309 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ef76d8_4829_4123_bbff_e7b36c59afba.slice/crio-0f45f32429dba607812dc922f7499da81f055ee505786c1ee866b7c391d5c613 WatchSource:0}: Error finding container 0f45f32429dba607812dc922f7499da81f055ee505786c1ee866b7c391d5c613: Status 404 returned error can't find the container with id 0f45f32429dba607812dc922f7499da81f055ee505786c1ee866b7c391d5c613 Oct 30 12:35:17.424098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:17.412462 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 12:35:17.424098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:17.412522 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" podUID="b7abde5f-37e2-40d0-9ddc-bd907e115aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 12:35:17.631497 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:17.631398 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" event={"ID":"35ef76d8-4829-4123-bbff-e7b36c59afba","Type":"ContainerStarted","Data":"6b96d53841725417ee5ab9a88eaf75c34a158c87944593ff48235f3d51639829"} Oct 30 12:35:17.631497 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:17.631459 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" event={"ID":"35ef76d8-4829-4123-bbff-e7b36c59afba","Type":"ContainerStarted","Data":"0f45f32429dba607812dc922f7499da81f055ee505786c1ee866b7c391d5c613"} Oct 30 12:35:17.631497 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:17.631490 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" Oct 30 12:35:17.678954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:17.678899 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" podStartSLOduration=1.678886298 podStartE2EDuration="1.678886298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:17.674667403 +0000 UTC m=+2468.708805810" watchObservedRunningTime="2026-10-30 12:35:17.678886298 +0000 UTC m=+2468.713024694" Oct 30 12:35:18.408096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:18.406443 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 12:35:18.408096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:18.406506 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" podUID="b7abde5f-37e2-40d0-9ddc-bd907e115aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 12:35:19.425092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:19.424838 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 12:35:19.425092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:19.424914 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" podUID="b7abde5f-37e2-40d0-9ddc-bd907e115aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 12:35:20.405402 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:20.405359 4468 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 12:35:20.405628 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:20.405442 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" podUID="b7abde5f-37e2-40d0-9ddc-bd907e115aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 12:35:21.492831 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:21.492745 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:21.493642 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:21.493611 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:21.494422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:21.494402 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:35:21.717153 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:21.717112 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx"] Oct 30 12:35:21.722133 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:21.722113 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" Oct 30 12:35:21.790140 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:21.786097 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-42220c\"" Oct 30 12:35:21.790140 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:21.786841 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-42220c-dockercfg-5m4qq\"" Oct 30 12:35:21.802163 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:21.802129 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdzz8\" (UniqueName: \"kubernetes.io/projected/e8e11991-76bb-4c45-8393-33dbc1ed877f-kube-api-access-pdzz8\") pod \"payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx\" (UID: \"e8e11991-76bb-4c45-8393-33dbc1ed877f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" Oct 30 12:35:21.802919 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:21.802893 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e11991-76bb-4c45-8393-33dbc1ed877f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx\" (UID: \"e8e11991-76bb-4c45-8393-33dbc1ed877f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" Oct 30 12:35:21.912690 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:21.912242 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdzz8\" (UniqueName: \"kubernetes.io/projected/e8e11991-76bb-4c45-8393-33dbc1ed877f-kube-api-access-pdzz8\") pod \"payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx\" (UID: \"e8e11991-76bb-4c45-8393-33dbc1ed877f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" Oct 30 12:35:21.912690 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:21.912339 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e11991-76bb-4c45-8393-33dbc1ed877f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx\" (UID: \"e8e11991-76bb-4c45-8393-33dbc1ed877f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" Oct 30 12:35:21.913473 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:21.913446 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e11991-76bb-4c45-8393-33dbc1ed877f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx\" (UID: \"e8e11991-76bb-4c45-8393-33dbc1ed877f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" Oct 30 12:35:21.927237 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:21.927203 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx"] Oct 30 12:35:21.948818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:21.948774 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdzz8\" (UniqueName: \"kubernetes.io/projected/e8e11991-76bb-4c45-8393-33dbc1ed877f-kube-api-access-pdzz8\") pod \"payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx\" (UID: \"e8e11991-76bb-4c45-8393-33dbc1ed877f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" Oct 30 12:35:22.216234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:22.216197 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" Oct 30 12:35:22.549400 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:22.549360 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx"] Oct 30 12:35:22.550738 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:35:22.550714 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e11991_76bb_4c45_8393_33dbc1ed877f.slice/crio-669a8bb959cfe1f4b46d9c8b7cc95cdf255ae90784e026d6f691ce314a3bf359 WatchSource:0}: Error finding container 669a8bb959cfe1f4b46d9c8b7cc95cdf255ae90784e026d6f691ce314a3bf359: Status 404 returned error can't find the container with id 669a8bb959cfe1f4b46d9c8b7cc95cdf255ae90784e026d6f691ce314a3bf359 Oct 30 12:35:22.672143 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:22.672096 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" event={"ID":"e8e11991-76bb-4c45-8393-33dbc1ed877f","Type":"ContainerStarted","Data":"dbcd4565c7d3b5508901ef4d453230a3daea9e259c786a25456848fab54fcf2c"} Oct 30 12:35:22.672305 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:22.672149 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" event={"ID":"e8e11991-76bb-4c45-8393-33dbc1ed877f","Type":"ContainerStarted","Data":"669a8bb959cfe1f4b46d9c8b7cc95cdf255ae90784e026d6f691ce314a3bf359"} Oct 30 12:35:22.672363 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:22.672345 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" Oct 30 12:35:22.717818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:22.717757 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" podStartSLOduration=1.717744178 podStartE2EDuration="1.717744178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:22.715035603 +0000 UTC m=+2473.749173998" watchObservedRunningTime="2026-10-30 12:35:22.717744178 +0000 UTC m=+2473.751882565" Oct 30 12:35:23.603198 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:35:23.603163 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:35:23.647754 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:23.647720 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" Oct 30 12:35:28.644849 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:28.644817 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" Oct 30 12:35:30.987693 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:30.987644 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g"] Oct 30 12:35:31.001625 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:31.001577 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g"] Oct 30 12:35:31.001941 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:31.001923 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g" Oct 30 12:35:31.038659 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:31.038626 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-dbe123\"" Oct 30 12:35:31.039013 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:31.038991 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-dbe123-dockercfg-ht8jb\"" Oct 30 12:35:31.115837 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:31.115798 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36dd06a2-cbb4-43df-8411-f3b35e06e98c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g\" (UID: \"36dd06a2-cbb4-43df-8411-f3b35e06e98c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g" Oct 30 12:35:31.116020 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:31.115866 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8spcq\" (UniqueName: \"kubernetes.io/projected/36dd06a2-cbb4-43df-8411-f3b35e06e98c-kube-api-access-8spcq\") pod \"payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g\" (UID: \"36dd06a2-cbb4-43df-8411-f3b35e06e98c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g" Oct 30 12:35:31.217871 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:31.217821 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36dd06a2-cbb4-43df-8411-f3b35e06e98c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g\" (UID: \"36dd06a2-cbb4-43df-8411-f3b35e06e98c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g" Oct 30 12:35:31.218129 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:31.217966 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8spcq\" (UniqueName: \"kubernetes.io/projected/36dd06a2-cbb4-43df-8411-f3b35e06e98c-kube-api-access-8spcq\") pod \"payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g\" (UID: \"36dd06a2-cbb4-43df-8411-f3b35e06e98c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g" Oct 30 12:35:31.218770 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:31.218748 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36dd06a2-cbb4-43df-8411-f3b35e06e98c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g\" (UID: \"36dd06a2-cbb4-43df-8411-f3b35e06e98c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g" Oct 30 12:35:31.242728 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:31.242674 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8spcq\" (UniqueName: \"kubernetes.io/projected/36dd06a2-cbb4-43df-8411-f3b35e06e98c-kube-api-access-8spcq\") pod \"payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g\" (UID: \"36dd06a2-cbb4-43df-8411-f3b35e06e98c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g" Oct 30 12:35:31.328032 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:31.327994 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g" Oct 30 12:35:31.639597 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:35:31.639556 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36dd06a2_cbb4_43df_8411_f3b35e06e98c.slice/crio-d2c0ad2b53b48db5b39ee589ea55eb1b285f114975d08412181837aa86c91fa5 WatchSource:0}: Error finding container d2c0ad2b53b48db5b39ee589ea55eb1b285f114975d08412181837aa86c91fa5: Status 404 returned error can't find the container with id d2c0ad2b53b48db5b39ee589ea55eb1b285f114975d08412181837aa86c91fa5 Oct 30 12:35:31.639851 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:31.639828 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g"] Oct 30 12:35:31.764723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:31.764253 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g" event={"ID":"36dd06a2-cbb4-43df-8411-f3b35e06e98c","Type":"ContainerStarted","Data":"d2c0ad2b53b48db5b39ee589ea55eb1b285f114975d08412181837aa86c91fa5"} Oct 30 12:35:32.463659 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:32.463628 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k"] Oct 30 12:35:32.467978 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:32.467950 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k" Oct 30 12:35:32.486784 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:32.486750 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k"] Oct 30 12:35:32.495553 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:32.495520 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-5d90d8-dockercfg-bfd7f\"" Oct 30 12:35:32.495721 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:32.495589 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-5d90d8\"" Oct 30 12:35:32.614153 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:32.542129 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pkph\" (UniqueName: \"kubernetes.io/projected/3a341ddc-19d0-47ff-9193-cafacc7fe172-kube-api-access-7pkph\") pod \"payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k\" (UID: \"3a341ddc-19d0-47ff-9193-cafacc7fe172\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k" Oct 30 12:35:32.685093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:32.667212 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a341ddc-19d0-47ff-9193-cafacc7fe172-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k\" (UID: \"3a341ddc-19d0-47ff-9193-cafacc7fe172\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k" Oct 30 12:35:32.802962 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:32.802880 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx"] Oct 30 12:35:32.817400 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:32.817375 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g" Oct 30 12:35:32.817551 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:32.817415 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx"] Oct 30 12:35:32.817551 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:32.817506 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx" Oct 30 12:35:32.817551 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:32.817516 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g" event={"ID":"36dd06a2-cbb4-43df-8411-f3b35e06e98c","Type":"ContainerStarted","Data":"4bd9410d575837a4c5f95cbb47157222a2806d5996f1531e0d1c26ef37b0456c"} Oct 30 12:35:32.833894 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:32.833869 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a341ddc-19d0-47ff-9193-cafacc7fe172-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k\" (UID: \"3a341ddc-19d0-47ff-9193-cafacc7fe172\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k" Oct 30 12:35:32.834014 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:32.834001 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pkph\" (UniqueName: \"kubernetes.io/projected/3a341ddc-19d0-47ff-9193-cafacc7fe172-kube-api-access-7pkph\") pod \"payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k\" (UID: \"3a341ddc-19d0-47ff-9193-cafacc7fe172\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k" Oct 30 12:35:32.834657 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:32.834632 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a341ddc-19d0-47ff-9193-cafacc7fe172-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k\" (UID: \"3a341ddc-19d0-47ff-9193-cafacc7fe172\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k" Oct 30 12:35:32.854205 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:32.854164 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g" podStartSLOduration=2.854153085 podStartE2EDuration="2.854153085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:32.852522343 +0000 UTC m=+2483.886660735" watchObservedRunningTime="2026-10-30 12:35:32.854153085 +0000 UTC m=+2483.888291492" Oct 30 12:35:32.860645 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:32.860622 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pkph\" (UniqueName: \"kubernetes.io/projected/3a341ddc-19d0-47ff-9193-cafacc7fe172-kube-api-access-7pkph\") pod \"payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k\" (UID: \"3a341ddc-19d0-47ff-9193-cafacc7fe172\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k" Oct 30 12:35:32.934854 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:32.934832 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1fde862-66cf-4df9-ab47-34adface139d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx\" (UID: \"c1fde862-66cf-4df9-ab47-34adface139d\") " pod="openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx" Oct 30 12:35:32.935095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:32.935048 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lpwn\" (UniqueName: \"kubernetes.io/projected/c1fde862-66cf-4df9-ab47-34adface139d-kube-api-access-2lpwn\") pod \"payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx\" (UID: \"c1fde862-66cf-4df9-ab47-34adface139d\") " pod="openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx" Oct 30 12:35:33.038917 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.038875 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lpwn\" (UniqueName: \"kubernetes.io/projected/c1fde862-66cf-4df9-ab47-34adface139d-kube-api-access-2lpwn\") pod \"payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx\" (UID: \"c1fde862-66cf-4df9-ab47-34adface139d\") " pod="openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx" Oct 30 12:35:33.039107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.039091 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1fde862-66cf-4df9-ab47-34adface139d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx\" (UID: \"c1fde862-66cf-4df9-ab47-34adface139d\") " pod="openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx" Oct 30 12:35:33.040190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.040163 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1fde862-66cf-4df9-ab47-34adface139d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx\" (UID: \"c1fde862-66cf-4df9-ab47-34adface139d\") " pod="openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx" Oct 30 12:35:33.063957 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.063894 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lpwn\" (UniqueName: \"kubernetes.io/projected/c1fde862-66cf-4df9-ab47-34adface139d-kube-api-access-2lpwn\") pod \"payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx\" (UID: \"c1fde862-66cf-4df9-ab47-34adface139d\") " pod="openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx" Oct 30 12:35:33.083116 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.083090 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k" Oct 30 12:35:33.132407 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.132372 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx" Oct 30 12:35:33.388890 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:35:33.388853 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a341ddc_19d0_47ff_9193_cafacc7fe172.slice/crio-2dec66823838ffe6c6f99ffa6b4111f0a7a0fb4e227dd0079f57c70aa488aaf4 WatchSource:0}: Error finding container 2dec66823838ffe6c6f99ffa6b4111f0a7a0fb4e227dd0079f57c70aa488aaf4: Status 404 returned error can't find the container with id 2dec66823838ffe6c6f99ffa6b4111f0a7a0fb4e227dd0079f57c70aa488aaf4 Oct 30 12:35:33.392051 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.391985 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k"] Oct 30 12:35:33.458029 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:35:33.458002 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fde862_66cf_4df9_ab47_34adface139d.slice/crio-f89aae1f0a4c6bea5e506bbe61a20b84d26d719e598516dd8139ecc6ef2f3b8c WatchSource:0}: Error finding container f89aae1f0a4c6bea5e506bbe61a20b84d26d719e598516dd8139ecc6ef2f3b8c: Status 404 returned error can't find the container with id f89aae1f0a4c6bea5e506bbe61a20b84d26d719e598516dd8139ecc6ef2f3b8c Oct 30 12:35:33.465248 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.465217 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx"] Oct 30 12:35:33.557652 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.557621 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb"] Oct 30 12:35:33.563049 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.563029 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" Oct 30 12:35:33.581913 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.581888 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-9d45fb-serving-cert\"" Oct 30 12:35:33.583151 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.583122 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb"] Oct 30 12:35:33.649750 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.649662 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pq9r\" (UniqueName: \"kubernetes.io/projected/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-kube-api-access-8pq9r\") pod \"maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb\" (UID: \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" Oct 30 12:35:33.649893 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.649760 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb\" (UID: \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" Oct 30 12:35:33.649893 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.649797 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb\" (UID: \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" Oct 30 12:35:33.649983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.649900 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb\" (UID: \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" Oct 30 12:35:33.681802 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.681782 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" Oct 30 12:35:33.751704 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.751662 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pq9r\" (UniqueName: \"kubernetes.io/projected/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-kube-api-access-8pq9r\") pod \"maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb\" (UID: \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" Oct 30 12:35:33.751885 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.751715 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb\" (UID: \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" Oct 30 12:35:33.751885 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.751738 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb\" (UID: \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" Oct 30 12:35:33.751885 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.751793 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb\" (UID: \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" Oct 30 12:35:33.752973 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.752942 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb\" (UID: \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" Oct 30 12:35:33.755114 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.755088 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb\" (UID: \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" Oct 30 12:35:33.755262 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.755135 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb\" (UID: \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" Oct 30 12:35:33.777383 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.777356 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pq9r\" (UniqueName: \"kubernetes.io/projected/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-kube-api-access-8pq9r\") pod \"maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb\" (UID: \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" Oct 30 12:35:33.816143 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.816114 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k" event={"ID":"3a341ddc-19d0-47ff-9193-cafacc7fe172","Type":"ContainerStarted","Data":"56172a53ae7649cc8c31962b8c3bef50c539a8dc8865d5a54fb50fe7eebc959c"} Oct 30 12:35:33.816241 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.816167 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k" event={"ID":"3a341ddc-19d0-47ff-9193-cafacc7fe172","Type":"ContainerStarted","Data":"2dec66823838ffe6c6f99ffa6b4111f0a7a0fb4e227dd0079f57c70aa488aaf4"} Oct 30 12:35:33.816326 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.816308 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k" Oct 30 12:35:33.818037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.818008 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx" event={"ID":"c1fde862-66cf-4df9-ab47-34adface139d","Type":"ContainerStarted","Data":"079d3a2ede10ca48951330f62bbda3d1fee5049bd24da05962592f6c03ae7c01"} Oct 30 12:35:33.818231 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.818207 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx" Oct 30 12:35:33.818307 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.818258 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx" event={"ID":"c1fde862-66cf-4df9-ab47-34adface139d","Type":"ContainerStarted","Data":"f89aae1f0a4c6bea5e506bbe61a20b84d26d719e598516dd8139ecc6ef2f3b8c"} Oct 30 12:35:33.891106 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.884729 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k" podStartSLOduration=1.884709343 podStartE2EDuration="1.884709343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:33.881498864 +0000 UTC m=+2484.915637260" watchObservedRunningTime="2026-10-30 12:35:33.884709343 +0000 UTC m=+2484.918847741" Oct 30 12:35:33.891106 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.890265 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" Oct 30 12:35:33.940537 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:33.940490 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx" podStartSLOduration=1.940474268 podStartE2EDuration="1.940474268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:33.937976733 +0000 UTC m=+2484.972115129" watchObservedRunningTime="2026-10-30 12:35:33.940474268 +0000 UTC m=+2484.974612664" Oct 30 12:35:34.238830 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:35:34.238796 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf2843b_7deb_4ffa_8b55_48e59ad1f756.slice/crio-ebf9c241a82f12a5dba81eef5e9e9a31a6921a2f1235cfe3e695d151b25dfe0c WatchSource:0}: Error finding container ebf9c241a82f12a5dba81eef5e9e9a31a6921a2f1235cfe3e695d151b25dfe0c: Status 404 returned error can't find the container with id ebf9c241a82f12a5dba81eef5e9e9a31a6921a2f1235cfe3e695d151b25dfe0c Oct 30 12:35:34.242171 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:34.242143 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb"] Oct 30 12:35:34.853063 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:34.853030 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" event={"ID":"9bf2843b-7deb-4ffa-8b55-48e59ad1f756","Type":"ContainerStarted","Data":"ebf9c241a82f12a5dba81eef5e9e9a31a6921a2f1235cfe3e695d151b25dfe0c"} Oct 30 12:35:35.859633 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:35.859601 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" event={"ID":"9bf2843b-7deb-4ffa-8b55-48e59ad1f756","Type":"ContainerStarted","Data":"1e560430c484a1cdc67aba5febed6504d78abe85beb4286c2783bc7cd9d87063"} Oct 30 12:35:35.860042 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:35.859659 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" Oct 30 12:35:35.908824 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:35.908776 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" podStartSLOduration=2.4931720139999998 podStartE2EDuration="2.908763922s" totalImagesPullingTime="415.591908ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:35.904504361 +0000 UTC m=+2486.938642756" watchObservedRunningTime="2026-10-30 12:35:35.908763922 +0000 UTC m=+2486.942902317" Oct 30 12:35:39.315797 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:39.315765 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm"] Oct 30 12:35:39.316242 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:39.315949 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" podUID="d37dfa55-ca61-425e-b32e-7b9846eadd70" containerName="istio-proxy" containerID="cri-o://bcd8a9707e902fc7f4d5588bc1a5226306420768daab801f9d9d2d6dbf19af56" gracePeriod=30 Oct 30 12:35:41.872141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:41.872111 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" Oct 30 12:35:43.821943 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:43.821911 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g" Oct 30 12:35:44.602266 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.602202 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:35:44.740625 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.740597 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-token\" (UniqueName: \"kubernetes.io/projected/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-token\") pod \"d37dfa55-ca61-425e-b32e-7b9846eadd70\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " Oct 30 12:35:44.740795 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.740637 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-podinfo\") pod \"d37dfa55-ca61-425e-b32e-7b9846eadd70\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " Oct 30 12:35:44.740795 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.740723 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-credential-socket\") pod \"d37dfa55-ca61-425e-b32e-7b9846eadd70\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " Oct 30 12:35:44.740795 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.740745 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-envoy\") pod \"d37dfa55-ca61-425e-b32e-7b9846eadd70\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " Oct 30 12:35:44.740795 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.740770 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-workload-socket\") pod \"d37dfa55-ca61-425e-b32e-7b9846eadd70\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " Oct 30 12:35:44.741059 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.740900 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-workload-certs\") pod \"d37dfa55-ca61-425e-b32e-7b9846eadd70\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " Oct 30 12:35:44.741059 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.740927 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d37dfa55-ca61-425e-b32e-7b9846eadd70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d37dfa55-ca61-425e-b32e-7b9846eadd70-istiod-ca-cert\") pod \"d37dfa55-ca61-425e-b32e-7b9846eadd70\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " Oct 30 12:35:44.741059 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.741032 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-data\") pod \"d37dfa55-ca61-425e-b32e-7b9846eadd70\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " Oct 30 12:35:44.741309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.741116 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37dfa55-ca61-425e-b32e-7b9846eadd70-kube-api-access-gqskj\" (UniqueName: \"kubernetes.io/projected/d37dfa55-ca61-425e-b32e-7b9846eadd70-kube-api-access-gqskj\") pod \"d37dfa55-ca61-425e-b32e-7b9846eadd70\" (UID: \"d37dfa55-ca61-425e-b32e-7b9846eadd70\") " Oct 30 12:35:44.741309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.741251 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-workload-socket" pod "d37dfa55-ca61-425e-b32e-7b9846eadd70" (UID: "d37dfa55-ca61-425e-b32e-7b9846eadd70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:35:44.741533 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.741505 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-workload-certs" pod "d37dfa55-ca61-425e-b32e-7b9846eadd70" (UID: "d37dfa55-ca61-425e-b32e-7b9846eadd70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:35:44.741617 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.741591 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-data" pod "d37dfa55-ca61-425e-b32e-7b9846eadd70" (UID: "d37dfa55-ca61-425e-b32e-7b9846eadd70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:35:44.741686 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.741610 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d37dfa55-ca61-425e-b32e-7b9846eadd70-istiod-ca-cert" pod "d37dfa55-ca61-425e-b32e-7b9846eadd70" (UID: "d37dfa55-ca61-425e-b32e-7b9846eadd70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:35:44.741801 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.741784 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:35:44.741862 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.741803 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:35:44.741862 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.741817 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d37dfa55-ca61-425e-b32e-7b9846eadd70-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:35:44.741862 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.741827 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:35:44.742027 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.741960 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-credential-socket" pod "d37dfa55-ca61-425e-b32e-7b9846eadd70" (UID: "d37dfa55-ca61-425e-b32e-7b9846eadd70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:35:44.743336 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.743306 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-envoy" pod "d37dfa55-ca61-425e-b32e-7b9846eadd70" (UID: "d37dfa55-ca61-425e-b32e-7b9846eadd70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:35:44.743594 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.743578 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-token" pod "d37dfa55-ca61-425e-b32e-7b9846eadd70" (UID: "d37dfa55-ca61-425e-b32e-7b9846eadd70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:35:44.743798 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.743783 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-podinfo" pod "d37dfa55-ca61-425e-b32e-7b9846eadd70" (UID: "d37dfa55-ca61-425e-b32e-7b9846eadd70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:35:44.743880 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.743830 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37dfa55-ca61-425e-b32e-7b9846eadd70-kube-api-access-gqskj" pod "d37dfa55-ca61-425e-b32e-7b9846eadd70" (UID: "d37dfa55-ca61-425e-b32e-7b9846eadd70"). InnerVolumeSpecName "kube-api-access-gqskj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:35:44.867142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.863192 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqskj\" (UniqueName: \"kubernetes.io/projected/d37dfa55-ca61-425e-b32e-7b9846eadd70-kube-api-access-gqskj\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:35:44.867142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.863225 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:35:44.867142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.863240 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:35:44.867142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.863251 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:35:44.867142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.863262 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d37dfa55-ca61-425e-b32e-7b9846eadd70-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:35:44.876098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.871233 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx" Oct 30 12:35:44.880115 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.879360 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k" Oct 30 12:35:44.984888 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.984831 4468 generic.go:356] "Generic (PLEG): container finished" podID="d37dfa55-ca61-425e-b32e-7b9846eadd70" containerID="bcd8a9707e902fc7f4d5588bc1a5226306420768daab801f9d9d2d6dbf19af56" exitCode=0 Oct 30 12:35:44.985591 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.984997 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" event={"ID":"d37dfa55-ca61-425e-b32e-7b9846eadd70","Type":"ContainerDied","Data":"bcd8a9707e902fc7f4d5588bc1a5226306420768daab801f9d9d2d6dbf19af56"} Oct 30 12:35:44.985591 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.985045 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" event={"ID":"d37dfa55-ca61-425e-b32e-7b9846eadd70","Type":"ContainerDied","Data":"51c629a5e30fe67b8a54f857abd38a697f8aec5389057f2e294fc23af9b4b63e"} Oct 30 12:35:44.985591 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.985122 4468 scope.go:122] "RemoveContainer" containerID="bcd8a9707e902fc7f4d5588bc1a5226306420768daab801f9d9d2d6dbf19af56" Oct 30 12:35:44.985591 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:44.985275 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm" Oct 30 12:35:45.020654 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:45.020303 4468 scope.go:122] "RemoveContainer" containerID="bcd8a9707e902fc7f4d5588bc1a5226306420768daab801f9d9d2d6dbf19af56" Oct 30 12:35:45.020842 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:35:45.020798 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcd8a9707e902fc7f4d5588bc1a5226306420768daab801f9d9d2d6dbf19af56\": container with ID starting with bcd8a9707e902fc7f4d5588bc1a5226306420768daab801f9d9d2d6dbf19af56 not found: ID does not exist" containerID="bcd8a9707e902fc7f4d5588bc1a5226306420768daab801f9d9d2d6dbf19af56" Oct 30 12:35:45.020909 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:45.020833 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd8a9707e902fc7f4d5588bc1a5226306420768daab801f9d9d2d6dbf19af56"} err="failed to get container status \"bcd8a9707e902fc7f4d5588bc1a5226306420768daab801f9d9d2d6dbf19af56\": rpc error: code = NotFound desc = could not find container \"bcd8a9707e902fc7f4d5588bc1a5226306420768daab801f9d9d2d6dbf19af56\": container with ID starting with bcd8a9707e902fc7f4d5588bc1a5226306420768daab801f9d9d2d6dbf19af56 not found: ID does not exist" Oct 30 12:35:45.054177 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:45.054145 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm"] Oct 30 12:35:45.058929 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:45.058899 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-2nghm"] Oct 30 12:35:45.092053 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:45.092022 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d37dfa55-ca61-425e-b32e-7b9846eadd70" path="/var/lib/kubelet/pods/d37dfa55-ca61-425e-b32e-7b9846eadd70/volumes" Oct 30 12:35:48.296769 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.296737 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5"] Oct 30 12:35:48.297411 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.297392 4468 cpu_manager.go:414] "containerMap: removing container" podUID="d37dfa55-ca61-425e-b32e-7b9846eadd70" containerName="istio-proxy" Oct 30 12:35:48.297411 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.297409 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37dfa55-ca61-425e-b32e-7b9846eadd70" containerName="istio-proxy" Oct 30 12:35:48.297533 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.297519 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37dfa55-ca61-425e-b32e-7b9846eadd70" containerName="istio-proxy" Oct 30 12:35:48.302783 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.302765 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" Oct 30 12:35:48.337496 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.337461 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-8d2170\"" Oct 30 12:35:48.338687 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.338666 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5"] Oct 30 12:35:48.374652 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.374624 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz"] Oct 30 12:35:48.379892 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.379871 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" Oct 30 12:35:48.389608 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.389589 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz"] Oct 30 12:35:48.396305 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.396280 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-8d2170-dockercfg-l6bwk\"" Oct 30 12:35:48.720091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.719895 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr4sr\" (UniqueName: \"kubernetes.io/projected/179f5f3b-6ed4-4330-91ee-6dcbf00ea87a-kube-api-access-zr4sr\") pod \"payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz\" (UID: \"179f5f3b-6ed4-4330-91ee-6dcbf00ea87a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" Oct 30 12:35:48.720091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.719964 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mw5n\" (UniqueName: \"kubernetes.io/projected/0022531e-0b86-46d8-b59e-ed23010efa07-kube-api-access-7mw5n\") pod \"payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5\" (UID: \"0022531e-0b86-46d8-b59e-ed23010efa07\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" Oct 30 12:35:48.720363 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.720118 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0022531e-0b86-46d8-b59e-ed23010efa07-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5\" (UID: \"0022531e-0b86-46d8-b59e-ed23010efa07\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" Oct 30 12:35:48.720363 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.720189 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/179f5f3b-6ed4-4330-91ee-6dcbf00ea87a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz\" (UID: \"179f5f3b-6ed4-4330-91ee-6dcbf00ea87a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" Oct 30 12:35:48.825092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.823291 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/179f5f3b-6ed4-4330-91ee-6dcbf00ea87a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz\" (UID: \"179f5f3b-6ed4-4330-91ee-6dcbf00ea87a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" Oct 30 12:35:48.825092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.823515 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr4sr\" (UniqueName: \"kubernetes.io/projected/179f5f3b-6ed4-4330-91ee-6dcbf00ea87a-kube-api-access-zr4sr\") pod \"payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz\" (UID: \"179f5f3b-6ed4-4330-91ee-6dcbf00ea87a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" Oct 30 12:35:48.825092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.823564 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mw5n\" (UniqueName: \"kubernetes.io/projected/0022531e-0b86-46d8-b59e-ed23010efa07-kube-api-access-7mw5n\") pod \"payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5\" (UID: \"0022531e-0b86-46d8-b59e-ed23010efa07\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" Oct 30 12:35:48.825092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.823647 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0022531e-0b86-46d8-b59e-ed23010efa07-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5\" (UID: \"0022531e-0b86-46d8-b59e-ed23010efa07\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" Oct 30 12:35:48.825092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.824652 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0022531e-0b86-46d8-b59e-ed23010efa07-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5\" (UID: \"0022531e-0b86-46d8-b59e-ed23010efa07\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" Oct 30 12:35:48.825543 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.825334 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/179f5f3b-6ed4-4330-91ee-6dcbf00ea87a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz\" (UID: \"179f5f3b-6ed4-4330-91ee-6dcbf00ea87a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" Oct 30 12:35:48.849578 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.849550 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr4sr\" (UniqueName: \"kubernetes.io/projected/179f5f3b-6ed4-4330-91ee-6dcbf00ea87a-kube-api-access-zr4sr\") pod \"payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz\" (UID: \"179f5f3b-6ed4-4330-91ee-6dcbf00ea87a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" Oct 30 12:35:48.849789 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.849756 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mw5n\" (UniqueName: \"kubernetes.io/projected/0022531e-0b86-46d8-b59e-ed23010efa07-kube-api-access-7mw5n\") pod \"payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5\" (UID: \"0022531e-0b86-46d8-b59e-ed23010efa07\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" Oct 30 12:35:48.945678 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.945644 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" Oct 30 12:35:48.996422 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:48.996343 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" Oct 30 12:35:49.278995 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:35:49.278966 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0022531e_0b86_46d8_b59e_ed23010efa07.slice/crio-073f338c05f8a7fb0a290aa9a245945668f8a6936d2eff6934fe717032dcfa71 WatchSource:0}: Error finding container 073f338c05f8a7fb0a290aa9a245945668f8a6936d2eff6934fe717032dcfa71: Status 404 returned error can't find the container with id 073f338c05f8a7fb0a290aa9a245945668f8a6936d2eff6934fe717032dcfa71 Oct 30 12:35:49.281375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:49.281350 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5"] Oct 30 12:35:49.432501 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:49.432467 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz"] Oct 30 12:35:49.433040 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:35:49.433013 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179f5f3b_6ed4_4330_91ee_6dcbf00ea87a.slice/crio-7f981b084e3a717d87207093f0718dc82c0a4aa6800587ecbbf9c74e12d74e37 WatchSource:0}: Error finding container 7f981b084e3a717d87207093f0718dc82c0a4aa6800587ecbbf9c74e12d74e37: Status 404 returned error can't find the container with id 7f981b084e3a717d87207093f0718dc82c0a4aa6800587ecbbf9c74e12d74e37 Oct 30 12:35:50.124194 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.124154 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" event={"ID":"179f5f3b-6ed4-4330-91ee-6dcbf00ea87a","Type":"ContainerStarted","Data":"8f5cde7c40b46ed60cf63473b3f046c56748a71abbd67c87cff7fea392e06d5b"} Oct 30 12:35:50.124500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.124472 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" event={"ID":"179f5f3b-6ed4-4330-91ee-6dcbf00ea87a","Type":"ContainerStarted","Data":"7f981b084e3a717d87207093f0718dc82c0a4aa6800587ecbbf9c74e12d74e37"} Oct 30 12:35:50.125726 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.125704 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" Oct 30 12:35:50.129910 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.129885 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" event={"ID":"0022531e-0b86-46d8-b59e-ed23010efa07","Type":"ContainerStarted","Data":"69c35757960310240363fde0857945765d07e5bfa89603294c2f3ed3f100cc7a"} Oct 30 12:35:50.130031 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.129923 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" event={"ID":"0022531e-0b86-46d8-b59e-ed23010efa07","Type":"ContainerStarted","Data":"073f338c05f8a7fb0a290aa9a245945668f8a6936d2eff6934fe717032dcfa71"} Oct 30 12:35:50.130031 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.130025 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" Oct 30 12:35:50.170786 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.170737 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" podStartSLOduration=2.17072318 podStartE2EDuration="2.17072318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:50.16845127 +0000 UTC m=+2501.202589665" watchObservedRunningTime="2026-10-30 12:35:50.17072318 +0000 UTC m=+2501.204861576" Oct 30 12:35:50.218859 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.218796 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" podStartSLOduration=2.218777805 podStartE2EDuration="2.218777805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:50.218571946 +0000 UTC m=+2501.252710343" watchObservedRunningTime="2026-10-30 12:35:50.218777805 +0000 UTC m=+2501.252916203" Oct 30 12:35:50.369089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.368734 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg"] Oct 30 12:35:50.394108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.394019 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg"] Oct 30 12:35:50.394428 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.394368 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" Oct 30 12:35:50.433970 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.433944 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-dbe123-serving-cert\"" Oct 30 12:35:50.477412 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.477372 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca7e7d8-d36f-4378-8052-04a62530fff6-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg\" (UID: \"fca7e7d8-d36f-4378-8052-04a62530fff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" Oct 30 12:35:50.477619 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.477598 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmkjd\" (UniqueName: \"kubernetes.io/projected/fca7e7d8-d36f-4378-8052-04a62530fff6-kube-api-access-nmkjd\") pod \"maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg\" (UID: \"fca7e7d8-d36f-4378-8052-04a62530fff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" Oct 30 12:35:50.477685 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.477643 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fca7e7d8-d36f-4378-8052-04a62530fff6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg\" (UID: \"fca7e7d8-d36f-4378-8052-04a62530fff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" Oct 30 12:35:50.477909 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.477884 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca7e7d8-d36f-4378-8052-04a62530fff6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg\" (UID: \"fca7e7d8-d36f-4378-8052-04a62530fff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" Oct 30 12:35:50.583094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.582039 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca7e7d8-d36f-4378-8052-04a62530fff6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg\" (UID: \"fca7e7d8-d36f-4378-8052-04a62530fff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" Oct 30 12:35:50.583094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.582169 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca7e7d8-d36f-4378-8052-04a62530fff6-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg\" (UID: \"fca7e7d8-d36f-4378-8052-04a62530fff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" Oct 30 12:35:50.583094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.582309 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmkjd\" (UniqueName: \"kubernetes.io/projected/fca7e7d8-d36f-4378-8052-04a62530fff6-kube-api-access-nmkjd\") pod \"maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg\" (UID: \"fca7e7d8-d36f-4378-8052-04a62530fff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" Oct 30 12:35:50.583094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.582348 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fca7e7d8-d36f-4378-8052-04a62530fff6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg\" (UID: \"fca7e7d8-d36f-4378-8052-04a62530fff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" Oct 30 12:35:50.583094 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:35:50.582630 4468 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-serving-cert: secret "maas-api-e2e-shared-b-w1-dbe123-serving-cert" not found Oct 30 12:35:50.583094 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:35:50.582725 4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fca7e7d8-d36f-4378-8052-04a62530fff6-maas-api-tls podName:fca7e7d8-d36f-4378-8052-04a62530fff6 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:51.082679595 +0000 UTC m=+2502.116817986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fca7e7d8-d36f-4378-8052-04a62530fff6-maas-api-tls") pod "maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" (UID: "fca7e7d8-d36f-4378-8052-04a62530fff6") : secret "maas-api-e2e-shared-b-w1-dbe123-serving-cert" not found Oct 30 12:35:50.593568 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.593538 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca7e7d8-d36f-4378-8052-04a62530fff6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg\" (UID: \"fca7e7d8-d36f-4378-8052-04a62530fff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" Oct 30 12:35:50.637097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.633884 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca7e7d8-d36f-4378-8052-04a62530fff6-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg\" (UID: \"fca7e7d8-d36f-4378-8052-04a62530fff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" Oct 30 12:35:50.655210 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:50.654400 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmkjd\" (UniqueName: \"kubernetes.io/projected/fca7e7d8-d36f-4378-8052-04a62530fff6-kube-api-access-nmkjd\") pod \"maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg\" (UID: \"fca7e7d8-d36f-4378-8052-04a62530fff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" Oct 30 12:35:51.090410 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:51.090377 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fca7e7d8-d36f-4378-8052-04a62530fff6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg\" (UID: \"fca7e7d8-d36f-4378-8052-04a62530fff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" Oct 30 12:35:51.092757 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:51.092729 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fca7e7d8-d36f-4378-8052-04a62530fff6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg\" (UID: \"fca7e7d8-d36f-4378-8052-04a62530fff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" Oct 30 12:35:51.338933 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:51.338904 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" Oct 30 12:35:51.652418 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:35:51.652374 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca7e7d8_d36f_4378_8052_04a62530fff6.slice/crio-cd33f193adce882466749d9e675890dfca0c071bc5b8bf312f9eb074fc5ac119 WatchSource:0}: Error finding container cd33f193adce882466749d9e675890dfca0c071bc5b8bf312f9eb074fc5ac119: Status 404 returned error can't find the container with id cd33f193adce882466749d9e675890dfca0c071bc5b8bf312f9eb074fc5ac119 Oct 30 12:35:51.654194 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:51.654173 4468 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:35:51.655395 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:51.655374 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg"] Oct 30 12:35:52.144829 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:52.144774 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" event={"ID":"fca7e7d8-d36f-4378-8052-04a62530fff6","Type":"ContainerStarted","Data":"29aec5190929b65b8dd107d4127220f7c04142e01ca78c0359c78972a18e21dd"} Oct 30 12:35:52.145000 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:52.144886 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" event={"ID":"fca7e7d8-d36f-4378-8052-04a62530fff6","Type":"ContainerStarted","Data":"cd33f193adce882466749d9e675890dfca0c071bc5b8bf312f9eb074fc5ac119"} Oct 30 12:35:52.145000 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:52.144941 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" Oct 30 12:35:52.198201 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:52.198148 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" podStartSLOduration=1.837134738 podStartE2EDuration="2.198134156s" totalImagesPullingTime="360.999418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:52.196669197 +0000 UTC m=+2503.230807593" watchObservedRunningTime="2026-10-30 12:35:52.198134156 +0000 UTC m=+2503.232272551" Oct 30 12:35:53.602340 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:35:53.602294 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:35:55.299245 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:55.299209 4468 patch_prober.go:29] interesting pod/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:35:55.300456 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:55.299267 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" podUID="4027e387-9043-4303-a6e1-c18365dcfac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:35:56.172469 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:56.172438 4468 generic.go:356] "Generic (PLEG): container finished" podID="4027e387-9043-4303-a6e1-c18365dcfac0" containerID="7ca63977ebf67617613e536e0fade02abaa1701a88ea870a25bc7000feb0d398" exitCode=0 Oct 30 12:35:56.172669 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:56.172513 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" event={"ID":"4027e387-9043-4303-a6e1-c18365dcfac0","Type":"ContainerDied","Data":"7ca63977ebf67617613e536e0fade02abaa1701a88ea870a25bc7000feb0d398"} Oct 30 12:35:56.172903 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:56.172885 4468 scope.go:122] "RemoveContainer" containerID="7ca63977ebf67617613e536e0fade02abaa1701a88ea870a25bc7000feb0d398" Oct 30 12:35:56.174452 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:56.174428 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:35:56.174519 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:56.174489 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:35:56.174560 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:56.174518 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:35:57.180321 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:57.180287 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" event={"ID":"4027e387-9043-4303-a6e1-c18365dcfac0","Type":"ContainerStarted","Data":"dd308d1b1bbc57bbb94ec515a3ecdb0ef44846bd3f779e1b52da259b1970a841"} Oct 30 12:35:58.076867 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:58.076831 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:35:58.077057 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:58.077034 4468 patch_prober.go:29] interesting pod/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 12:35:58.077147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:58.077118 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" podUID="4027e387-9043-4303-a6e1-c18365dcfac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 12:35:58.157905 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:58.157870 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" Oct 30 12:35:59.077807 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:59.077772 4468 patch_prober.go:29] interesting pod/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 12:35:59.078319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:35:59.077826 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" podUID="4027e387-9043-4303-a6e1-c18365dcfac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 12:36:00.077414 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:00.077381 4468 patch_prober.go:29] interesting pod/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 12:36:00.077603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:00.077452 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" podUID="4027e387-9043-4303-a6e1-c18365dcfac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 12:36:01.081668 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:01.081584 4468 patch_prober.go:29] interesting pod/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 12:36:01.096590 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:01.096536 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" podUID="4027e387-9043-4303-a6e1-c18365dcfac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 12:36:01.142910 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:01.142869 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" Oct 30 12:36:02.077958 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:02.077922 4468 patch_prober.go:29] interesting pod/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 12:36:02.078167 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:02.077999 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" podUID="4027e387-9043-4303-a6e1-c18365dcfac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 12:36:02.148743 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:02.148703 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" Oct 30 12:36:03.129092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:03.128794 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:36:03.131193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:03.129500 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:36:03.131729 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:03.131708 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 12:36:10.499773 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.499727 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk"] Oct 30 12:36:10.506919 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.506205 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.539980 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.539947 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk"] Oct 30 12:36:10.678746 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.678706 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.678922 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.678816 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51b18829-f11f-406e-99b0-decf56caaf67-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.678922 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.678874 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn2gq\" (UniqueName: \"kubernetes.io/projected/51b18829-f11f-406e-99b0-decf56caaf67-kube-api-access-tn2gq\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.679053 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.678979 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b18829-f11f-406e-99b0-decf56caaf67-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.679053 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.679021 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b18829-f11f-406e-99b0-decf56caaf67-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.679194 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.679178 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.679255 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.679211 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.679909 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.679884 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.680011 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.679955 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.782132 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.782012 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.782132 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.782112 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.782380 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.782226 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.782380 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.782289 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51b18829-f11f-406e-99b0-decf56caaf67-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.782380 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.782361 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn2gq\" (UniqueName: \"kubernetes.io/projected/51b18829-f11f-406e-99b0-decf56caaf67-kube-api-access-tn2gq\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.782523 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.782395 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b18829-f11f-406e-99b0-decf56caaf67-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.782523 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.782438 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b18829-f11f-406e-99b0-decf56caaf67-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.782608 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.782549 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.782608 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.782588 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.782708 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.782606 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.783304 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.783275 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.783456 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.783351 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.783524 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.783464 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.783846 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.783819 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b18829-f11f-406e-99b0-decf56caaf67-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.785706 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.785685 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.786027 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.786004 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b18829-f11f-406e-99b0-decf56caaf67-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.808340 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.808309 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51b18829-f11f-406e-99b0-decf56caaf67-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.808474 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.808430 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn2gq\" (UniqueName: \"kubernetes.io/projected/51b18829-f11f-406e-99b0-decf56caaf67-kube-api-access-tn2gq\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:10.843883 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:10.843856 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:11.162783 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:36:11.162753 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b18829_f11f_406e_99b0_decf56caaf67.slice/crio-95551f310c000b465c6faf564740d29e02575c63d3769e2dfd48df12b34fee95 WatchSource:0}: Error finding container 95551f310c000b465c6faf564740d29e02575c63d3769e2dfd48df12b34fee95: Status 404 returned error can't find the container with id 95551f310c000b465c6faf564740d29e02575c63d3769e2dfd48df12b34fee95 Oct 30 12:36:11.165542 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:11.165506 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:36:11.165659 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:11.165576 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:36:11.165659 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:11.165606 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:36:11.165952 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:11.165929 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk"] Oct 30 12:36:11.332940 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:11.332903 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" event={"ID":"51b18829-f11f-406e-99b0-decf56caaf67","Type":"ContainerStarted","Data":"6c568aaabbeecaf1aeb70fe46d455e5a96a406f86f51a3f0a9521a01f147cdf2"} Oct 30 12:36:11.333149 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:11.332964 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" event={"ID":"51b18829-f11f-406e-99b0-decf56caaf67","Type":"ContainerStarted","Data":"95551f310c000b465c6faf564740d29e02575c63d3769e2dfd48df12b34fee95"} Oct 30 12:36:11.334866 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:11.334842 4468 generic.go:356] "Generic (PLEG): container finished" podID="73e2eab8-2195-41e0-97a7-63383ccbbe48" containerID="454ccb1c17bbc3e0b94fc856c34c9b6f5ce6efcd14552010891686d4a0fc55b2" exitCode=0 Oct 30 12:36:11.334961 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:11.334894 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" event={"ID":"73e2eab8-2195-41e0-97a7-63383ccbbe48","Type":"ContainerDied","Data":"454ccb1c17bbc3e0b94fc856c34c9b6f5ce6efcd14552010891686d4a0fc55b2"} Oct 30 12:36:11.335938 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:11.335910 4468 scope.go:122] "RemoveContainer" containerID="454ccb1c17bbc3e0b94fc856c34c9b6f5ce6efcd14552010891686d4a0fc55b2" Oct 30 12:36:11.342085 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:11.339687 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:36:11.342085 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:11.339846 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:36:11.342085 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:11.339939 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:36:11.385458 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:11.385402 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" podStartSLOduration=1.385383351 podStartE2EDuration="1.385383351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:11.380433421 +0000 UTC m=+2522.414571817" watchObservedRunningTime="2026-10-30 12:36:11.385383351 +0000 UTC m=+2522.419521748" Oct 30 12:36:11.815135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:11.815108 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz"] Oct 30 12:36:11.818524 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:11.818505 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" Oct 30 12:36:11.844514 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:11.844471 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz"] Oct 30 12:36:11.844729 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:11.844558 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:11.857383 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:11.857350 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-8d2170-serving-cert\"" Oct 30 12:36:12.012095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.008160 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29260bd5-b3f6-4df0-8917-573986f5969a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz\" (UID: \"29260bd5-b3f6-4df0-8917-573986f5969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" Oct 30 12:36:12.012095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.008260 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29260bd5-b3f6-4df0-8917-573986f5969a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz\" (UID: \"29260bd5-b3f6-4df0-8917-573986f5969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" Oct 30 12:36:12.012095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.008444 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29260bd5-b3f6-4df0-8917-573986f5969a-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz\" (UID: \"29260bd5-b3f6-4df0-8917-573986f5969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" Oct 30 12:36:12.012095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.008474 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd52c\" (UniqueName: \"kubernetes.io/projected/29260bd5-b3f6-4df0-8917-573986f5969a-kube-api-access-dd52c\") pod \"maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz\" (UID: \"29260bd5-b3f6-4df0-8917-573986f5969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" Oct 30 12:36:12.112147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.111324 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29260bd5-b3f6-4df0-8917-573986f5969a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz\" (UID: \"29260bd5-b3f6-4df0-8917-573986f5969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" Oct 30 12:36:12.112147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.111435 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29260bd5-b3f6-4df0-8917-573986f5969a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz\" (UID: \"29260bd5-b3f6-4df0-8917-573986f5969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" Oct 30 12:36:12.112147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.111684 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29260bd5-b3f6-4df0-8917-573986f5969a-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz\" (UID: \"29260bd5-b3f6-4df0-8917-573986f5969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" Oct 30 12:36:12.112147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.111731 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd52c\" (UniqueName: \"kubernetes.io/projected/29260bd5-b3f6-4df0-8917-573986f5969a-kube-api-access-dd52c\") pod \"maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz\" (UID: \"29260bd5-b3f6-4df0-8917-573986f5969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" Oct 30 12:36:12.112522 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:36:12.112504 4468 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-serving-cert: secret "maas-api-e2e-shared-b-w4-8d2170-serving-cert" not found Oct 30 12:36:12.121918 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:36:12.112584 4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29260bd5-b3f6-4df0-8917-573986f5969a-maas-api-tls podName:29260bd5-b3f6-4df0-8917-573986f5969a nodeName:}" failed. No retries permitted until 2026-10-30 12:36:12.612549473 +0000 UTC m=+2523.646687861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29260bd5-b3f6-4df0-8917-573986f5969a-maas-api-tls") pod "maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" (UID: "29260bd5-b3f6-4df0-8917-573986f5969a") : secret "maas-api-e2e-shared-b-w4-8d2170-serving-cert" not found Oct 30 12:36:12.121918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.116694 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29260bd5-b3f6-4df0-8917-573986f5969a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz\" (UID: \"29260bd5-b3f6-4df0-8917-573986f5969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" Oct 30 12:36:12.142983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.127832 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29260bd5-b3f6-4df0-8917-573986f5969a-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz\" (UID: \"29260bd5-b3f6-4df0-8917-573986f5969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" Oct 30 12:36:12.143781 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.143746 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd52c\" (UniqueName: \"kubernetes.io/projected/29260bd5-b3f6-4df0-8917-573986f5969a-kube-api-access-dd52c\") pod \"maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz\" (UID: \"29260bd5-b3f6-4df0-8917-573986f5969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" Oct 30 12:36:12.361146 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.361097 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" event={"ID":"73e2eab8-2195-41e0-97a7-63383ccbbe48","Type":"ContainerStarted","Data":"c5c50ce3d3ce81f5687ab0de35881a9ca9075703802cc431f0f0bf62127e30c4"} Oct 30 12:36:12.611094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.608229 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:36:12.615093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.613383 4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 12:36:12.615093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.613480 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" podUID="73e2eab8-2195-41e0-97a7-63383ccbbe48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 12:36:12.661093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.660822 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29260bd5-b3f6-4df0-8917-573986f5969a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz\" (UID: \"29260bd5-b3f6-4df0-8917-573986f5969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" Oct 30 12:36:12.668892 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.668848 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29260bd5-b3f6-4df0-8917-573986f5969a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz\" (UID: \"29260bd5-b3f6-4df0-8917-573986f5969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" Oct 30 12:36:12.739097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.735398 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" Oct 30 12:36:12.860126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.859260 4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 12:36:12.860126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:12.859340 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" podUID="51b18829-f11f-406e-99b0-decf56caaf67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 12:36:13.396162 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:36:13.396126 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29260bd5_b3f6_4df0_8917_573986f5969a.slice/crio-2c604f45dcf744148703fc0160c8928cb5456a6c08ca8a6d0d1b3928a8f12a0e WatchSource:0}: Error finding container 2c604f45dcf744148703fc0160c8928cb5456a6c08ca8a6d0d1b3928a8f12a0e: Status 404 returned error can't find the container with id 2c604f45dcf744148703fc0160c8928cb5456a6c08ca8a6d0d1b3928a8f12a0e Oct 30 12:36:13.400160 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:13.400134 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz"] Oct 30 12:36:13.530114 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:13.528961 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:36:13.611092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:13.607592 4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 12:36:13.611092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:13.607659 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" podUID="73e2eab8-2195-41e0-97a7-63383ccbbe48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 12:36:13.846027 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:13.844237 4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 12:36:13.846027 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:13.844305 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" podUID="51b18829-f11f-406e-99b0-decf56caaf67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 12:36:14.378639 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:14.378599 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" event={"ID":"29260bd5-b3f6-4df0-8917-573986f5969a","Type":"ContainerStarted","Data":"34688e5e1f01d298f22e7078a90f26c76046d19c244e750653625484e9d2f78e"} Oct 30 12:36:14.379157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:14.378666 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" event={"ID":"29260bd5-b3f6-4df0-8917-573986f5969a","Type":"ContainerStarted","Data":"2c604f45dcf744148703fc0160c8928cb5456a6c08ca8a6d0d1b3928a8f12a0e"} Oct 30 12:36:14.379157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:14.378698 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" Oct 30 12:36:14.434101 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:14.432485 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" podStartSLOduration=3.039125085 podStartE2EDuration="3.432463139s" totalImagesPullingTime="393.338054ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:14.429524018 +0000 UTC m=+2525.463662414" watchObservedRunningTime="2026-10-30 12:36:14.432463139 +0000 UTC m=+2525.466601537" Oct 30 12:36:14.611562 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:14.611523 4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 12:36:14.611777 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:14.611589 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" podUID="73e2eab8-2195-41e0-97a7-63383ccbbe48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 12:36:14.846093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:14.845474 4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 12:36:14.846093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:14.845537 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" podUID="51b18829-f11f-406e-99b0-decf56caaf67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 12:36:15.627096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:15.624563 4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 12:36:15.627096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:15.624630 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" podUID="73e2eab8-2195-41e0-97a7-63383ccbbe48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 12:36:15.846098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:15.844507 4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 12:36:15.846098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:15.844568 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" podUID="51b18829-f11f-406e-99b0-decf56caaf67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 12:36:16.634148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:16.629518 4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 12:36:16.634148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:16.629583 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" podUID="73e2eab8-2195-41e0-97a7-63383ccbbe48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 12:36:16.845094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:16.844511 4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 12:36:16.845094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:16.844578 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" podUID="51b18829-f11f-406e-99b0-decf56caaf67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 12:36:17.613108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:17.612561 4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 12:36:17.613108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:17.612634 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" podUID="73e2eab8-2195-41e0-97a7-63383ccbbe48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 12:36:17.850030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:17.849994 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:17.850628 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:17.850369 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:17.851080 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:17.851044 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:36:18.610509 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:18.610475 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:36:18.611656 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:18.611636 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:36:20.391766 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:20.391735 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" Oct 30 12:36:23.601337 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:36:23.601298 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:36:53.601233 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:36:53.601147 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:36:55.613700 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.613659 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r"] Oct 30 12:36:55.617716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.617693 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.657364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.657321 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r"] Oct 30 12:36:55.684316 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.684265 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.684536 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.684329 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.684536 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.684441 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.684536 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.684474 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.684536 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.684509 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcmzj\" (UniqueName: \"kubernetes.io/projected/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-kube-api-access-dcmzj\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.684694 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.684666 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.684749 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.684721 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.684836 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.684816 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.684889 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.684878 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.790145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.786451 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.790145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.786517 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.790145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.786557 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.790145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.786733 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.790145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.786920 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.790145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.787001 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.790145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.787042 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.790145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.787100 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcmzj\" (UniqueName: \"kubernetes.io/projected/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-kube-api-access-dcmzj\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.790145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.787234 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.790145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.788698 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.790145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.789857 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.790145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.790156 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.790944 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.790447 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.790944 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.790767 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.800130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.800035 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.809632 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.808529 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.824058 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.824026 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.827784 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.827752 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcmzj\" (UniqueName: \"kubernetes.io/projected/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-kube-api-access-dcmzj\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:55.934342 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:55.934306 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:56.259652 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:36:56.259624 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77483d1_fb3c_4be6_8d49_c6863bfcc84d.slice/crio-0ce08c13dd33b4a0f23e9421ec666bbaf2fc58760a6e2f4d4980cd9d496abdf3 WatchSource:0}: Error finding container 0ce08c13dd33b4a0f23e9421ec666bbaf2fc58760a6e2f4d4980cd9d496abdf3: Status 404 returned error can't find the container with id 0ce08c13dd33b4a0f23e9421ec666bbaf2fc58760a6e2f4d4980cd9d496abdf3 Oct 30 12:36:56.262836 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:56.262797 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:36:56.262942 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:56.262873 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r"] Oct 30 12:36:56.262942 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:56.262880 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:36:56.263112 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:56.262960 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:36:56.784793 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:56.784742 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" event={"ID":"b77483d1-fb3c-4be6-8d49-c6863bfcc84d","Type":"ContainerStarted","Data":"4034c93f561b1ef45372c69924d5859fc11ad2be3b45f227d1eed9c1f21e5184"} Oct 30 12:36:56.785324 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:56.784806 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" event={"ID":"b77483d1-fb3c-4be6-8d49-c6863bfcc84d","Type":"ContainerStarted","Data":"0ce08c13dd33b4a0f23e9421ec666bbaf2fc58760a6e2f4d4980cd9d496abdf3"} Oct 30 12:36:56.849290 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:56.849232 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" podStartSLOduration=1.8492174700000001 podStartE2EDuration="1.84921747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:56.846691852 +0000 UTC m=+2567.880830249" watchObservedRunningTime="2026-10-30 12:36:56.84921747 +0000 UTC m=+2567.883355865" Oct 30 12:36:56.935340 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:56.935306 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:36:57.936822 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:57.936784 4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 12:36:57.937359 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:57.936851 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" podUID="b77483d1-fb3c-4be6-8d49-c6863bfcc84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 12:36:58.935680 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:58.935645 4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 12:36:58.935886 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:58.935703 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" podUID="b77483d1-fb3c-4be6-8d49-c6863bfcc84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 12:36:59.934934 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:59.934898 4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 12:36:59.935420 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:36:59.934957 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" podUID="b77483d1-fb3c-4be6-8d49-c6863bfcc84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 12:37:00.935669 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:00.935630 4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 12:37:00.936100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:00.935692 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" podUID="b77483d1-fb3c-4be6-8d49-c6863bfcc84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 12:37:01.934753 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:01.934705 4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 12:37:01.934946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:01.934795 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" podUID="b77483d1-fb3c-4be6-8d49-c6863bfcc84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 12:37:02.940351 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:02.940315 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:37:02.940754 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:02.940613 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:37:02.941465 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:02.941444 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 12:37:10.694562 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:10.694526 4468 scope.go:122] "RemoveContainer" containerID="9e069b97fb837265211d9a108dd146852423275bfafc118e8e799d4cddf0a4ce" Oct 30 12:37:18.018903 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.018869 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw"] Oct 30 12:37:18.022992 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.022970 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.041629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.041599 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw"] Oct 30 12:37:18.049981 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.049954 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-42220c\"/\"auth-test-model-42220c-kserve-self-signed-certs\"" Oct 30 12:37:18.052517 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.052480 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-42220c\"/\"kube-root-ca.crt\"" Oct 30 12:37:18.052517 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.052502 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-42220c\"/\"default-dockercfg-s8rlh\"" Oct 30 12:37:18.052690 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.052606 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-42220c\"/\"openshift-service-ca.crt\"" Oct 30 12:37:18.113405 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.113373 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-dshm\") pod \"auth-test-model-42220c-kserve-7667b48d65-9srkw\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.113619 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.113601 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-home\") pod \"auth-test-model-42220c-kserve-7667b48d65-9srkw\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.113698 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.113632 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-tmp-dir\") pod \"auth-test-model-42220c-kserve-7667b48d65-9srkw\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.113698 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.113667 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-model-cache\") pod \"auth-test-model-42220c-kserve-7667b48d65-9srkw\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.113786 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.113767 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffsnn\" (UniqueName: \"kubernetes.io/projected/1c00f519-078c-4b00-bd7f-a758e916a650-kube-api-access-ffsnn\") pod \"auth-test-model-42220c-kserve-7667b48d65-9srkw\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.113824 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.113810 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c00f519-078c-4b00-bd7f-a758e916a650-tls-certs\") pod \"auth-test-model-42220c-kserve-7667b48d65-9srkw\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.215450 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.215418 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffsnn\" (UniqueName: \"kubernetes.io/projected/1c00f519-078c-4b00-bd7f-a758e916a650-kube-api-access-ffsnn\") pod \"auth-test-model-42220c-kserve-7667b48d65-9srkw\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.215642 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.215559 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c00f519-078c-4b00-bd7f-a758e916a650-tls-certs\") pod \"auth-test-model-42220c-kserve-7667b48d65-9srkw\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.215642 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.215610 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-dshm\") pod \"auth-test-model-42220c-kserve-7667b48d65-9srkw\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.215847 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.215827 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-home\") pod \"auth-test-model-42220c-kserve-7667b48d65-9srkw\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.215926 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.215867 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-tmp-dir\") pod \"auth-test-model-42220c-kserve-7667b48d65-9srkw\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.215926 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.215910 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-model-cache\") pod \"auth-test-model-42220c-kserve-7667b48d65-9srkw\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.216262 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.216234 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-home\") pod \"auth-test-model-42220c-kserve-7667b48d65-9srkw\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.216346 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.216288 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-tmp-dir\") pod \"auth-test-model-42220c-kserve-7667b48d65-9srkw\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.216386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.216372 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-model-cache\") pod \"auth-test-model-42220c-kserve-7667b48d65-9srkw\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.217938 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.217911 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-dshm\") pod \"auth-test-model-42220c-kserve-7667b48d65-9srkw\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.218230 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.218214 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c00f519-078c-4b00-bd7f-a758e916a650-tls-certs\") pod \"auth-test-model-42220c-kserve-7667b48d65-9srkw\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.281140 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.277181 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffsnn\" (UniqueName: \"kubernetes.io/projected/1c00f519-078c-4b00-bd7f-a758e916a650-kube-api-access-ffsnn\") pod \"auth-test-model-42220c-kserve-7667b48d65-9srkw\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.347171 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.346342 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:18.671222 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:37:18.671186 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c00f519_078c_4b00_bd7f_a758e916a650.slice/crio-4c212be1cd695812b1a19064bf9603de03b6699eb25539946eace4268dbebdc2 WatchSource:0}: Error finding container 4c212be1cd695812b1a19064bf9603de03b6699eb25539946eace4268dbebdc2: Status 404 returned error can't find the container with id 4c212be1cd695812b1a19064bf9603de03b6699eb25539946eace4268dbebdc2 Oct 30 12:37:18.674473 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:18.674448 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw"] Oct 30 12:37:19.061734 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:19.061630 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" event={"ID":"1c00f519-078c-4b00-bd7f-a758e916a650","Type":"ContainerStarted","Data":"6773647367c2da889f45659d8b5475439e5a42e8f567aaa81482fb2a6c2c3e33"} Oct 30 12:37:19.061734 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:19.061684 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" event={"ID":"1c00f519-078c-4b00-bd7f-a758e916a650","Type":"ContainerStarted","Data":"4c212be1cd695812b1a19064bf9603de03b6699eb25539946eace4268dbebdc2"} Oct 30 12:37:19.107923 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:19.107869 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" podStartSLOduration=2.107857842 podStartE2EDuration="2.107857842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:19.106432438 +0000 UTC m=+2590.140570833" watchObservedRunningTime="2026-10-30 12:37:19.107857842 +0000 UTC m=+2590.141996238" Oct 30 12:37:19.346835 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:19.346757 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:23.600109 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:37:23.600058 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:37:25.178337 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:25.178297 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx"] Oct 30 12:37:25.212811 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:25.212767 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k"] Oct 30 12:37:28.347249 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:28.347186 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:28.359470 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:28.359445 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:29.141633 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:29.141601 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:37:30.178892 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:30.178848 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx" podUID="c1fde862-66cf-4df9-ab47-34adface139d" containerName="payload-processing" containerID="cri-o://079d3a2ede10ca48951330f62bbda3d1fee5049bd24da05962592f6c03ae7c01" gracePeriod=25 Oct 30 12:37:30.213949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:30.213894 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k" podUID="3a341ddc-19d0-47ff-9193-cafacc7fe172" containerName="payload-pre-processing" containerID="cri-o://56172a53ae7649cc8c31962b8c3bef50c539a8dc8865d5a54fb50fe7eebc959c" gracePeriod=25 Oct 30 12:37:30.492157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:30.492129 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx" Oct 30 12:37:30.535791 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:30.535761 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1fde862-66cf-4df9-ab47-34adface139d-kube-api-access-2lpwn\" (UniqueName: \"kubernetes.io/projected/c1fde862-66cf-4df9-ab47-34adface139d-kube-api-access-2lpwn\") pod \"c1fde862-66cf-4df9-ab47-34adface139d\" (UID: \"c1fde862-66cf-4df9-ab47-34adface139d\") " Oct 30 12:37:30.535963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:30.535945 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1fde862-66cf-4df9-ab47-34adface139d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1fde862-66cf-4df9-ab47-34adface139d-plugins-config-volume\") pod \"c1fde862-66cf-4df9-ab47-34adface139d\" (UID: \"c1fde862-66cf-4df9-ab47-34adface139d\") " Oct 30 12:37:30.536864 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:30.536802 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1fde862-66cf-4df9-ab47-34adface139d-plugins-config-volume" pod "c1fde862-66cf-4df9-ab47-34adface139d" (UID: "c1fde862-66cf-4df9-ab47-34adface139d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:37:30.537009 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:30.536971 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1fde862-66cf-4df9-ab47-34adface139d-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:37:30.538675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:30.538651 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1fde862-66cf-4df9-ab47-34adface139d-kube-api-access-2lpwn" pod "c1fde862-66cf-4df9-ab47-34adface139d" (UID: "c1fde862-66cf-4df9-ab47-34adface139d"). InnerVolumeSpecName "kube-api-access-2lpwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:37:30.562109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:30.557855 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k" Oct 30 12:37:30.638002 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:30.637959 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a341ddc-19d0-47ff-9193-cafacc7fe172-kube-api-access-7pkph\" (UniqueName: \"kubernetes.io/projected/3a341ddc-19d0-47ff-9193-cafacc7fe172-kube-api-access-7pkph\") pod \"3a341ddc-19d0-47ff-9193-cafacc7fe172\" (UID: \"3a341ddc-19d0-47ff-9193-cafacc7fe172\") " Oct 30 12:37:30.638257 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:30.638148 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a341ddc-19d0-47ff-9193-cafacc7fe172-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a341ddc-19d0-47ff-9193-cafacc7fe172-plugins-config-volume\") pod \"3a341ddc-19d0-47ff-9193-cafacc7fe172\" (UID: \"3a341ddc-19d0-47ff-9193-cafacc7fe172\") " Oct 30 12:37:30.638761 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:30.638730 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a341ddc-19d0-47ff-9193-cafacc7fe172-plugins-config-volume" pod "3a341ddc-19d0-47ff-9193-cafacc7fe172" (UID: "3a341ddc-19d0-47ff-9193-cafacc7fe172"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:37:30.638880 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:30.638861 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a341ddc-19d0-47ff-9193-cafacc7fe172-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:37:30.638946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:30.638882 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lpwn\" (UniqueName: \"kubernetes.io/projected/c1fde862-66cf-4df9-ab47-34adface139d-kube-api-access-2lpwn\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:37:30.641020 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:30.640993 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a341ddc-19d0-47ff-9193-cafacc7fe172-kube-api-access-7pkph" pod "3a341ddc-19d0-47ff-9193-cafacc7fe172" (UID: "3a341ddc-19d0-47ff-9193-cafacc7fe172"). InnerVolumeSpecName "kube-api-access-7pkph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:37:30.740135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:30.740023 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pkph\" (UniqueName: \"kubernetes.io/projected/3a341ddc-19d0-47ff-9193-cafacc7fe172-kube-api-access-7pkph\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:37:31.145765 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:31.145731 4468 generic.go:356] "Generic (PLEG): container finished" podID="c1fde862-66cf-4df9-ab47-34adface139d" containerID="079d3a2ede10ca48951330f62bbda3d1fee5049bd24da05962592f6c03ae7c01" exitCode=0 Oct 30 12:37:31.145953 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:31.145783 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx" Oct 30 12:37:31.145953 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:31.145808 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx" event={"ID":"c1fde862-66cf-4df9-ab47-34adface139d","Type":"ContainerDied","Data":"079d3a2ede10ca48951330f62bbda3d1fee5049bd24da05962592f6c03ae7c01"} Oct 30 12:37:31.145953 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:31.145864 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx" event={"ID":"c1fde862-66cf-4df9-ab47-34adface139d","Type":"ContainerDied","Data":"f89aae1f0a4c6bea5e506bbe61a20b84d26d719e598516dd8139ecc6ef2f3b8c"} Oct 30 12:37:31.145953 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:31.145891 4468 scope.go:122] "RemoveContainer" containerID="079d3a2ede10ca48951330f62bbda3d1fee5049bd24da05962592f6c03ae7c01" Oct 30 12:37:31.147585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:31.147560 4468 generic.go:356] "Generic (PLEG): container finished" podID="3a341ddc-19d0-47ff-9193-cafacc7fe172" containerID="56172a53ae7649cc8c31962b8c3bef50c539a8dc8865d5a54fb50fe7eebc959c" exitCode=0 Oct 30 12:37:31.147721 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:31.147615 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k" Oct 30 12:37:31.147721 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:31.147636 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k" event={"ID":"3a341ddc-19d0-47ff-9193-cafacc7fe172","Type":"ContainerDied","Data":"56172a53ae7649cc8c31962b8c3bef50c539a8dc8865d5a54fb50fe7eebc959c"} Oct 30 12:37:31.147721 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:31.147675 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k" event={"ID":"3a341ddc-19d0-47ff-9193-cafacc7fe172","Type":"ContainerDied","Data":"2dec66823838ffe6c6f99ffa6b4111f0a7a0fb4e227dd0079f57c70aa488aaf4"} Oct 30 12:37:31.157298 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:31.157278 4468 scope.go:122] "RemoveContainer" containerID="079d3a2ede10ca48951330f62bbda3d1fee5049bd24da05962592f6c03ae7c01" Oct 30 12:37:31.157592 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:37:31.157572 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"079d3a2ede10ca48951330f62bbda3d1fee5049bd24da05962592f6c03ae7c01\": container with ID starting with 079d3a2ede10ca48951330f62bbda3d1fee5049bd24da05962592f6c03ae7c01 not found: ID does not exist" containerID="079d3a2ede10ca48951330f62bbda3d1fee5049bd24da05962592f6c03ae7c01" Oct 30 12:37:31.157675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:31.157602 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"079d3a2ede10ca48951330f62bbda3d1fee5049bd24da05962592f6c03ae7c01"} err="failed to get container status \"079d3a2ede10ca48951330f62bbda3d1fee5049bd24da05962592f6c03ae7c01\": rpc error: code = NotFound desc = could not find container \"079d3a2ede10ca48951330f62bbda3d1fee5049bd24da05962592f6c03ae7c01\": container with ID starting with 079d3a2ede10ca48951330f62bbda3d1fee5049bd24da05962592f6c03ae7c01 not found: ID does not exist" Oct 30 12:37:31.157675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:31.157618 4468 scope.go:122] "RemoveContainer" containerID="56172a53ae7649cc8c31962b8c3bef50c539a8dc8865d5a54fb50fe7eebc959c" Oct 30 12:37:31.166973 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:31.166960 4468 scope.go:122] "RemoveContainer" containerID="56172a53ae7649cc8c31962b8c3bef50c539a8dc8865d5a54fb50fe7eebc959c" Oct 30 12:37:31.167251 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:37:31.167232 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56172a53ae7649cc8c31962b8c3bef50c539a8dc8865d5a54fb50fe7eebc959c\": container with ID starting with 56172a53ae7649cc8c31962b8c3bef50c539a8dc8865d5a54fb50fe7eebc959c not found: ID does not exist" containerID="56172a53ae7649cc8c31962b8c3bef50c539a8dc8865d5a54fb50fe7eebc959c" Oct 30 12:37:31.167312 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:31.167258 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56172a53ae7649cc8c31962b8c3bef50c539a8dc8865d5a54fb50fe7eebc959c"} err="failed to get container status \"56172a53ae7649cc8c31962b8c3bef50c539a8dc8865d5a54fb50fe7eebc959c\": rpc error: code = NotFound desc = could not find container \"56172a53ae7649cc8c31962b8c3bef50c539a8dc8865d5a54fb50fe7eebc959c\": container with ID starting with 56172a53ae7649cc8c31962b8c3bef50c539a8dc8865d5a54fb50fe7eebc959c not found: ID does not exist" Oct 30 12:37:31.203287 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:31.203256 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx"] Oct 30 12:37:31.209509 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:31.209482 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5d90d8-84f9d86b66-v45sx"] Oct 30 12:37:31.245323 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:31.245293 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k"] Oct 30 12:37:31.249656 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:31.249634 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5d90d8-56c8565499-6568k"] Oct 30 12:37:33.093044 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:33.093002 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a341ddc-19d0-47ff-9193-cafacc7fe172" path="/var/lib/kubelet/pods/3a341ddc-19d0-47ff-9193-cafacc7fe172/volumes" Oct 30 12:37:33.093595 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:33.093516 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1fde862-66cf-4df9-ab47-34adface139d" path="/var/lib/kubelet/pods/c1fde862-66cf-4df9-ab47-34adface139d/volumes" Oct 30 12:37:47.170547 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.170512 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv"] Oct 30 12:37:47.171292 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.171272 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3a341ddc-19d0-47ff-9193-cafacc7fe172" containerName="payload-pre-processing" Oct 30 12:37:47.171372 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.171292 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a341ddc-19d0-47ff-9193-cafacc7fe172" containerName="payload-pre-processing" Oct 30 12:37:47.171372 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.171309 4468 cpu_manager.go:414] "containerMap: removing container" podUID="c1fde862-66cf-4df9-ab47-34adface139d" containerName="payload-processing" Oct 30 12:37:47.171372 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.171314 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1fde862-66cf-4df9-ab47-34adface139d" containerName="payload-processing" Oct 30 12:37:47.171542 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.171411 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1fde862-66cf-4df9-ab47-34adface139d" containerName="payload-processing" Oct 30 12:37:47.171542 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.171429 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a341ddc-19d0-47ff-9193-cafacc7fe172" containerName="payload-pre-processing" Oct 30 12:37:47.175566 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.175546 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.197214 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.197185 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv"] Oct 30 12:37:47.203919 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.203896 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-e1a321-openshift-default-dockercfg-l6xmw\"" Oct 30 12:37:47.306095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.294055 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-data\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.306561 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.306527 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-workload-socket\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.306757 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.306738 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-credential-socket\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.329094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.315186 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-token\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.329094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.315239 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhdtq\" (UniqueName: \"kubernetes.io/projected/b99555a5-b740-4eda-b8c4-30e2c3162c2c-kube-api-access-dhdtq\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.329094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.315330 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istiod-ca-cert\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.329094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.315492 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-podinfo\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.329094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.315519 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-envoy\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.329094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.315554 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-workload-certs\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.435202 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.432888 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-workload-socket\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.435202 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.432975 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-credential-socket\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.435202 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.433017 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-token\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.435202 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.433048 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhdtq\" (UniqueName: \"kubernetes.io/projected/b99555a5-b740-4eda-b8c4-30e2c3162c2c-kube-api-access-dhdtq\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.435202 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.433143 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istiod-ca-cert\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.435202 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.433288 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-podinfo\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.435202 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.433318 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-envoy\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.435202 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.433350 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-workload-certs\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.435202 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.433438 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-data\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.435202 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.434016 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-data\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.435202 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.434514 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-workload-socket\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.435202 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.434730 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-credential-socket\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.447162 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.436326 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istiod-ca-cert\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.447162 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.436906 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-workload-certs\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.484526 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.452952 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-envoy\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.496039 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.491662 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-podinfo\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.513728 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.513692 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-token\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.525091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.522476 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhdtq\" (UniqueName: \"kubernetes.io/projected/b99555a5-b740-4eda-b8c4-30e2c3162c2c-kube-api-access-dhdtq\") pod \"e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:47.790706 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:47.790603 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:48.126653 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:37:48.126610 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99555a5_b740_4eda_b8c4_30e2c3162c2c.slice/crio-ce7152e8265f41dc2727e50dcfbdcc0f6e6f1ae80a734fcfbab7ab56987db330 WatchSource:0}: Error finding container ce7152e8265f41dc2727e50dcfbdcc0f6e6f1ae80a734fcfbab7ab56987db330: Status 404 returned error can't find the container with id ce7152e8265f41dc2727e50dcfbdcc0f6e6f1ae80a734fcfbab7ab56987db330 Oct 30 12:37:48.128475 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:48.128441 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv"] Oct 30 12:37:48.129386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:48.129356 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:37:48.129445 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:48.129419 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:37:48.129487 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:48.129449 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:37:48.386448 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:48.386347 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" event={"ID":"b99555a5-b740-4eda-b8c4-30e2c3162c2c","Type":"ContainerStarted","Data":"3c46d01ecb3dc989c464d7b72717770c23da43660808876d6ae6b3b7d92960bc"} Oct 30 12:37:48.386448 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:48.386421 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" event={"ID":"b99555a5-b740-4eda-b8c4-30e2c3162c2c","Type":"ContainerStarted","Data":"ce7152e8265f41dc2727e50dcfbdcc0f6e6f1ae80a734fcfbab7ab56987db330"} Oct 30 12:37:48.439169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:48.439105 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" podStartSLOduration=1.439084148 podStartE2EDuration="1.439084148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:48.433135206 +0000 UTC m=+2619.467273598" watchObservedRunningTime="2026-10-30 12:37:48.439084148 +0000 UTC m=+2619.473222549" Oct 30 12:37:48.791415 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:48.791378 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:49.798144 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:49.798113 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:50.399695 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:50.399667 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:50.400587 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:37:50.400564 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:37:53.624673 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:37:53.624636 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:38:18.308091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:18.299326 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv"] Oct 30 12:38:18.308091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:18.299612 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" podUID="b99555a5-b740-4eda-b8c4-30e2c3162c2c" containerName="istio-proxy" containerID="cri-o://3c46d01ecb3dc989c464d7b72717770c23da43660808876d6ae6b3b7d92960bc" gracePeriod=30 Oct 30 12:38:20.401915 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:20.401877 4468 patch_prober.go:29] interesting pod/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:38:20.402390 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:20.401941 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" podUID="b99555a5-b740-4eda-b8c4-30e2c3162c2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:38:23.602951 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:38:23.602906 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:38:23.705119 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.705087 4468 generic.go:356] "Generic (PLEG): container finished" podID="b99555a5-b740-4eda-b8c4-30e2c3162c2c" containerID="3c46d01ecb3dc989c464d7b72717770c23da43660808876d6ae6b3b7d92960bc" exitCode=0 Oct 30 12:38:23.705429 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.705143 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" event={"ID":"b99555a5-b740-4eda-b8c4-30e2c3162c2c","Type":"ContainerDied","Data":"3c46d01ecb3dc989c464d7b72717770c23da43660808876d6ae6b3b7d92960bc"} Oct 30 12:38:23.845650 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.845619 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:38:23.881718 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.881670 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-envoy\") pod \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " Oct 30 12:38:23.881919 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.881766 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istiod-ca-cert\") pod \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " Oct 30 12:38:23.881919 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.881843 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-workload-socket\") pod \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " Oct 30 12:38:23.881919 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.881886 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-workload-certs\") pod \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " Oct 30 12:38:23.882106 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.881952 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-podinfo\") pod \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " Oct 30 12:38:23.882106 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.881989 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99555a5-b740-4eda-b8c4-30e2c3162c2c-kube-api-access-dhdtq\" (UniqueName: \"kubernetes.io/projected/b99555a5-b740-4eda-b8c4-30e2c3162c2c-kube-api-access-dhdtq\") pod \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " Oct 30 12:38:23.882220 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.882138 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-workload-socket" pod "b99555a5-b740-4eda-b8c4-30e2c3162c2c" (UID: "b99555a5-b740-4eda-b8c4-30e2c3162c2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:38:23.882220 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.882152 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-data\") pod \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " Oct 30 12:38:23.882329 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.882265 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-workload-certs" pod "b99555a5-b740-4eda-b8c4-30e2c3162c2c" (UID: "b99555a5-b740-4eda-b8c4-30e2c3162c2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:38:23.882329 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.882275 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-token\" (UniqueName: \"kubernetes.io/projected/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-token\") pod \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " Oct 30 12:38:23.882445 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.882378 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-data" pod "b99555a5-b740-4eda-b8c4-30e2c3162c2c" (UID: "b99555a5-b740-4eda-b8c4-30e2c3162c2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:38:23.882445 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.882427 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-credential-socket\") pod \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\" (UID: \"b99555a5-b740-4eda-b8c4-30e2c3162c2c\") " Oct 30 12:38:23.882818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.882782 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-credential-socket" pod "b99555a5-b740-4eda-b8c4-30e2c3162c2c" (UID: "b99555a5-b740-4eda-b8c4-30e2c3162c2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:38:23.882818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.882807 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istiod-ca-cert" pod "b99555a5-b740-4eda-b8c4-30e2c3162c2c" (UID: "b99555a5-b740-4eda-b8c4-30e2c3162c2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:38:23.883238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.883212 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:38:23.883341 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.883242 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:38:23.883341 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.883257 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:38:23.883341 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.883271 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:38:23.883341 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.883285 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:38:23.884530 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.884500 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-envoy" pod "b99555a5-b740-4eda-b8c4-30e2c3162c2c" (UID: "b99555a5-b740-4eda-b8c4-30e2c3162c2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:38:23.884530 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.884517 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-token" pod "b99555a5-b740-4eda-b8c4-30e2c3162c2c" (UID: "b99555a5-b740-4eda-b8c4-30e2c3162c2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:38:23.884654 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.884623 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99555a5-b740-4eda-b8c4-30e2c3162c2c-kube-api-access-dhdtq" pod "b99555a5-b740-4eda-b8c4-30e2c3162c2c" (UID: "b99555a5-b740-4eda-b8c4-30e2c3162c2c"). InnerVolumeSpecName "kube-api-access-dhdtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:38:23.884776 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.884758 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-podinfo" pod "b99555a5-b740-4eda-b8c4-30e2c3162c2c" (UID: "b99555a5-b740-4eda-b8c4-30e2c3162c2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:38:23.986252 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.986220 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:38:23.986712 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.986690 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhdtq\" (UniqueName: \"kubernetes.io/projected/b99555a5-b740-4eda-b8c4-30e2c3162c2c-kube-api-access-dhdtq\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:38:23.986838 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.986819 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:38:23.986901 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:23.986841 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99555a5-b740-4eda-b8c4-30e2c3162c2c-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:38:24.715523 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:24.715489 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" event={"ID":"b99555a5-b740-4eda-b8c4-30e2c3162c2c","Type":"ContainerDied","Data":"ce7152e8265f41dc2727e50dcfbdcc0f6e6f1ae80a734fcfbab7ab56987db330"} Oct 30 12:38:24.715981 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:24.715535 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv" Oct 30 12:38:24.715981 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:24.715552 4468 scope.go:122] "RemoveContainer" containerID="3c46d01ecb3dc989c464d7b72717770c23da43660808876d6ae6b3b7d92960bc" Oct 30 12:38:24.787510 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:24.787473 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv"] Oct 30 12:38:24.801153 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:24.801112 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-e1a321-openshift-default-6ccc876f6c-zqqrv"] Oct 30 12:38:25.091947 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:38:25.091870 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b99555a5-b740-4eda-b8c4-30e2c3162c2c" path="/var/lib/kubelet/pods/b99555a5-b740-4eda-b8c4-30e2c3162c2c/volumes" Oct 30 12:38:53.599653 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:38:53.599610 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:39:12.628440 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:12.628403 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5"] Oct 30 12:39:12.629095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:12.629079 4468 cpu_manager.go:414] "containerMap: removing container" podUID="b99555a5-b740-4eda-b8c4-30e2c3162c2c" containerName="istio-proxy" Oct 30 12:39:12.629147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:12.629095 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99555a5-b740-4eda-b8c4-30e2c3162c2c" containerName="istio-proxy" Oct 30 12:39:12.629247 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:12.629237 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99555a5-b740-4eda-b8c4-30e2c3162c2c" containerName="istio-proxy" Oct 30 12:39:12.634189 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:12.634166 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5" Oct 30 12:39:12.654609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:12.654573 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5"] Oct 30 12:39:12.660371 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:12.660343 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-73bafc\"" Oct 30 12:39:12.660531 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:12.660414 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-73bafc-dockercfg-mqbv4\"" Oct 30 12:39:12.748097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:12.747142 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t6vh\" (UniqueName: \"kubernetes.io/projected/6bac85e8-7e06-4933-a757-4739e81a0afd-kube-api-access-4t6vh\") pod \"payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5\" (UID: \"6bac85e8-7e06-4933-a757-4739e81a0afd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5" Oct 30 12:39:12.748097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:12.747242 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bac85e8-7e06-4933-a757-4739e81a0afd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5\" (UID: \"6bac85e8-7e06-4933-a757-4739e81a0afd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5" Oct 30 12:39:12.856034 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:12.855991 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t6vh\" (UniqueName: \"kubernetes.io/projected/6bac85e8-7e06-4933-a757-4739e81a0afd-kube-api-access-4t6vh\") pod \"payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5\" (UID: \"6bac85e8-7e06-4933-a757-4739e81a0afd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5" Oct 30 12:39:12.856403 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:12.856375 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bac85e8-7e06-4933-a757-4739e81a0afd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5\" (UID: \"6bac85e8-7e06-4933-a757-4739e81a0afd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5" Oct 30 12:39:12.858298 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:12.858176 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bac85e8-7e06-4933-a757-4739e81a0afd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5\" (UID: \"6bac85e8-7e06-4933-a757-4739e81a0afd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5" Oct 30 12:39:12.922312 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:12.922257 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t6vh\" (UniqueName: \"kubernetes.io/projected/6bac85e8-7e06-4933-a757-4739e81a0afd-kube-api-access-4t6vh\") pod \"payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5\" (UID: \"6bac85e8-7e06-4933-a757-4739e81a0afd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5" Oct 30 12:39:12.948691 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:12.948659 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5" Oct 30 12:39:13.268480 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:39:13.268451 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bac85e8_7e06_4933_a757_4739e81a0afd.slice/crio-1482e5bff9b04d34f93f992e2386922101c518dae0edbe2a181c78492c0e21f7 WatchSource:0}: Error finding container 1482e5bff9b04d34f93f992e2386922101c518dae0edbe2a181c78492c0e21f7: Status 404 returned error can't find the container with id 1482e5bff9b04d34f93f992e2386922101c518dae0edbe2a181c78492c0e21f7 Oct 30 12:39:13.271256 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:13.271231 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5"] Oct 30 12:39:14.091519 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:14.091480 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5" event={"ID":"6bac85e8-7e06-4933-a757-4739e81a0afd","Type":"ContainerStarted","Data":"40007ecdeeb66519502bf7ec525b445c6cc72f317346485fbceab5312d4d707a"} Oct 30 12:39:14.091919 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:14.091547 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5" event={"ID":"6bac85e8-7e06-4933-a757-4739e81a0afd","Type":"ContainerStarted","Data":"1482e5bff9b04d34f93f992e2386922101c518dae0edbe2a181c78492c0e21f7"} Oct 30 12:39:14.091919 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:14.091577 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5" Oct 30 12:39:14.159954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:14.159901 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5" podStartSLOduration=2.159885827 podStartE2EDuration="2.159885827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:39:14.134131169 +0000 UTC m=+2705.168269565" watchObservedRunningTime="2026-10-30 12:39:14.159885827 +0000 UTC m=+2705.194024223" Oct 30 12:39:23.636724 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:39:23.636682 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:39:25.110484 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:25.110437 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5" Oct 30 12:39:33.131283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:33.131248 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v"] Oct 30 12:39:33.135949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:33.135927 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" Oct 30 12:39:33.157815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:33.157777 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v"] Oct 30 12:39:33.163504 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:33.163476 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-73bafc-serving-cert\"" Oct 30 12:39:33.183043 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:33.183006 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/557e9c97-61e1-4358-8249-c79d946c5a9c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v\" (UID: \"557e9c97-61e1-4358-8249-c79d946c5a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" Oct 30 12:39:33.183253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:33.183064 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/557e9c97-61e1-4358-8249-c79d946c5a9c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v\" (UID: \"557e9c97-61e1-4358-8249-c79d946c5a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" Oct 30 12:39:33.183253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:33.183174 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48q7t\" (UniqueName: \"kubernetes.io/projected/557e9c97-61e1-4358-8249-c79d946c5a9c-kube-api-access-48q7t\") pod \"maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v\" (UID: \"557e9c97-61e1-4358-8249-c79d946c5a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" Oct 30 12:39:33.183412 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:33.183391 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/557e9c97-61e1-4358-8249-c79d946c5a9c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v\" (UID: \"557e9c97-61e1-4358-8249-c79d946c5a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" Oct 30 12:39:33.284900 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:33.284857 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/557e9c97-61e1-4358-8249-c79d946c5a9c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v\" (UID: \"557e9c97-61e1-4358-8249-c79d946c5a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" Oct 30 12:39:33.285108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:33.285029 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/557e9c97-61e1-4358-8249-c79d946c5a9c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v\" (UID: \"557e9c97-61e1-4358-8249-c79d946c5a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" Oct 30 12:39:33.285108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:33.285059 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/557e9c97-61e1-4358-8249-c79d946c5a9c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v\" (UID: \"557e9c97-61e1-4358-8249-c79d946c5a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" Oct 30 12:39:33.285108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:33.285104 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48q7t\" (UniqueName: \"kubernetes.io/projected/557e9c97-61e1-4358-8249-c79d946c5a9c-kube-api-access-48q7t\") pod \"maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v\" (UID: \"557e9c97-61e1-4358-8249-c79d946c5a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" Oct 30 12:39:33.285835 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:33.285805 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/557e9c97-61e1-4358-8249-c79d946c5a9c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v\" (UID: \"557e9c97-61e1-4358-8249-c79d946c5a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" Oct 30 12:39:33.287509 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:33.287481 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/557e9c97-61e1-4358-8249-c79d946c5a9c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v\" (UID: \"557e9c97-61e1-4358-8249-c79d946c5a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" Oct 30 12:39:33.287611 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:33.287573 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/557e9c97-61e1-4358-8249-c79d946c5a9c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v\" (UID: \"557e9c97-61e1-4358-8249-c79d946c5a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" Oct 30 12:39:33.309741 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:33.309715 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48q7t\" (UniqueName: \"kubernetes.io/projected/557e9c97-61e1-4358-8249-c79d946c5a9c-kube-api-access-48q7t\") pod \"maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v\" (UID: \"557e9c97-61e1-4358-8249-c79d946c5a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" Oct 30 12:39:33.457872 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:33.457581 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" Oct 30 12:39:33.770222 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:39:33.770165 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557e9c97_61e1_4358_8249_c79d946c5a9c.slice/crio-129a4ee6321885d684a6b7eb49f40b66f337b27d7f6f5c43361990404aea7a29 WatchSource:0}: Error finding container 129a4ee6321885d684a6b7eb49f40b66f337b27d7f6f5c43361990404aea7a29: Status 404 returned error can't find the container with id 129a4ee6321885d684a6b7eb49f40b66f337b27d7f6f5c43361990404aea7a29 Oct 30 12:39:33.773684 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:33.773655 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v"] Oct 30 12:39:34.329826 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:34.329790 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" event={"ID":"557e9c97-61e1-4358-8249-c79d946c5a9c","Type":"ContainerStarted","Data":"129a4ee6321885d684a6b7eb49f40b66f337b27d7f6f5c43361990404aea7a29"} Oct 30 12:39:35.338763 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:35.338723 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" event={"ID":"557e9c97-61e1-4358-8249-c79d946c5a9c","Type":"ContainerStarted","Data":"6f989341796d1760cb1bef6c31ccf07cb626d2ccd509224c25160d203403752c"} Oct 30 12:39:35.339219 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:35.338795 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" Oct 30 12:39:35.388169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:35.388115 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" podStartSLOduration=1.97781217 podStartE2EDuration="2.388099709s" totalImagesPullingTime="410.287539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:39:35.383498689 +0000 UTC m=+2726.417637082" watchObservedRunningTime="2026-10-30 12:39:35.388099709 +0000 UTC m=+2726.422238109" Oct 30 12:39:41.366140 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:39:41.366052 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" Oct 30 12:39:53.599430 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:39:53.599336 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:40:07.839120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:07.839062 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm"] Oct 30 12:40:07.848020 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:07.846562 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" podUID="87e88604-3549-4999-aefa-9bccc1042c87" containerName="istio-proxy" containerID="cri-o://c5ed3d61a9e7025b96bc43fde405c331a545680968771a4ec5ade6895b292df3" gracePeriod=30 Oct 30 12:40:13.152674 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.152632 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:40:13.336799 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.336765 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-istio-data\") pod \"87e88604-3549-4999-aefa-9bccc1042c87\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " Oct 30 12:40:13.337015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.336869 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e88604-3549-4999-aefa-9bccc1042c87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e88604-3549-4999-aefa-9bccc1042c87-istiod-ca-cert\") pod \"87e88604-3549-4999-aefa-9bccc1042c87\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " Oct 30 12:40:13.337015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.336925 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e88604-3549-4999-aefa-9bccc1042c87-kube-api-access-b9gzl\" (UniqueName: \"kubernetes.io/projected/87e88604-3549-4999-aefa-9bccc1042c87-kube-api-access-b9gzl\") pod \"87e88604-3549-4999-aefa-9bccc1042c87\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " Oct 30 12:40:13.337015 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.336960 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-workload-certs\") pod \"87e88604-3549-4999-aefa-9bccc1042c87\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " Oct 30 12:40:13.337240 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.337030 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-istio-envoy\") pod \"87e88604-3549-4999-aefa-9bccc1042c87\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " Oct 30 12:40:13.337240 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.337063 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e88604-3549-4999-aefa-9bccc1042c87-istio-token\" (UniqueName: \"kubernetes.io/projected/87e88604-3549-4999-aefa-9bccc1042c87-istio-token\") pod \"87e88604-3549-4999-aefa-9bccc1042c87\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " Oct 30 12:40:13.337240 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.337144 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-istio-data" pod "87e88604-3549-4999-aefa-9bccc1042c87" (UID: "87e88604-3549-4999-aefa-9bccc1042c87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:13.337240 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.337151 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-credential-socket\") pod \"87e88604-3549-4999-aefa-9bccc1042c87\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " Oct 30 12:40:13.337446 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.337266 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87e88604-3549-4999-aefa-9bccc1042c87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e88604-3549-4999-aefa-9bccc1042c87-istio-podinfo\") pod \"87e88604-3549-4999-aefa-9bccc1042c87\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " Oct 30 12:40:13.337446 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.337353 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-workload-socket\") pod \"87e88604-3549-4999-aefa-9bccc1042c87\" (UID: \"87e88604-3549-4999-aefa-9bccc1042c87\") " Oct 30 12:40:13.337568 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.337474 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-workload-certs" pod "87e88604-3549-4999-aefa-9bccc1042c87" (UID: "87e88604-3549-4999-aefa-9bccc1042c87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:13.337568 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.337477 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e88604-3549-4999-aefa-9bccc1042c87-istiod-ca-cert" pod "87e88604-3549-4999-aefa-9bccc1042c87" (UID: "87e88604-3549-4999-aefa-9bccc1042c87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:40:13.338061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.338034 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-credential-socket" pod "87e88604-3549-4999-aefa-9bccc1042c87" (UID: "87e88604-3549-4999-aefa-9bccc1042c87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:13.338443 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.338286 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:13.338597 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.338577 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e88604-3549-4999-aefa-9bccc1042c87-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:13.338597 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.338598 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:13.338789 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.338525 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-workload-socket" pod "87e88604-3549-4999-aefa-9bccc1042c87" (UID: "87e88604-3549-4999-aefa-9bccc1042c87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:13.340413 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.340389 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e88604-3549-4999-aefa-9bccc1042c87-istio-token" pod "87e88604-3549-4999-aefa-9bccc1042c87" (UID: "87e88604-3549-4999-aefa-9bccc1042c87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:13.340518 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.340498 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e88604-3549-4999-aefa-9bccc1042c87-kube-api-access-b9gzl" pod "87e88604-3549-4999-aefa-9bccc1042c87" (UID: "87e88604-3549-4999-aefa-9bccc1042c87"). InnerVolumeSpecName "kube-api-access-b9gzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:13.340606 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.340534 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87e88604-3549-4999-aefa-9bccc1042c87-istio-podinfo" pod "87e88604-3549-4999-aefa-9bccc1042c87" (UID: "87e88604-3549-4999-aefa-9bccc1042c87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:40:13.340731 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.340717 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-istio-envoy" pod "87e88604-3549-4999-aefa-9bccc1042c87" (UID: "87e88604-3549-4999-aefa-9bccc1042c87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:13.439557 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.439525 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:13.439557 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.439552 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e88604-3549-4999-aefa-9bccc1042c87-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:13.439557 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.439563 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:13.439809 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.439572 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e88604-3549-4999-aefa-9bccc1042c87-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:13.439809 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.439581 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e88604-3549-4999-aefa-9bccc1042c87-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:13.439809 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.439589 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9gzl\" (UniqueName: \"kubernetes.io/projected/87e88604-3549-4999-aefa-9bccc1042c87-kube-api-access-b9gzl\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:13.626832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.626795 4468 generic.go:356] "Generic (PLEG): container finished" podID="87e88604-3549-4999-aefa-9bccc1042c87" containerID="c5ed3d61a9e7025b96bc43fde405c331a545680968771a4ec5ade6895b292df3" exitCode=0 Oct 30 12:40:13.627008 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.626839 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" Oct 30 12:40:13.627008 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.626879 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" event={"ID":"87e88604-3549-4999-aefa-9bccc1042c87","Type":"ContainerDied","Data":"c5ed3d61a9e7025b96bc43fde405c331a545680968771a4ec5ade6895b292df3"} Oct 30 12:40:13.627008 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.626936 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm" event={"ID":"87e88604-3549-4999-aefa-9bccc1042c87","Type":"ContainerDied","Data":"1332fe13ae79948b4d9c9f9b6c9e0d59049a550fac10f2d1db8a09453b0bb19a"} Oct 30 12:40:13.627008 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.626963 4468 scope.go:122] "RemoveContainer" containerID="c5ed3d61a9e7025b96bc43fde405c331a545680968771a4ec5ade6895b292df3" Oct 30 12:40:13.638420 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.638402 4468 scope.go:122] "RemoveContainer" containerID="c5ed3d61a9e7025b96bc43fde405c331a545680968771a4ec5ade6895b292df3" Oct 30 12:40:13.638700 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:40:13.638682 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ed3d61a9e7025b96bc43fde405c331a545680968771a4ec5ade6895b292df3\": container with ID starting with c5ed3d61a9e7025b96bc43fde405c331a545680968771a4ec5ade6895b292df3 not found: ID does not exist" containerID="c5ed3d61a9e7025b96bc43fde405c331a545680968771a4ec5ade6895b292df3" Oct 30 12:40:13.638746 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.638709 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ed3d61a9e7025b96bc43fde405c331a545680968771a4ec5ade6895b292df3"} err="failed to get container status \"c5ed3d61a9e7025b96bc43fde405c331a545680968771a4ec5ade6895b292df3\": rpc error: code = NotFound desc = could not find container \"c5ed3d61a9e7025b96bc43fde405c331a545680968771a4ec5ade6895b292df3\": container with ID starting with c5ed3d61a9e7025b96bc43fde405c331a545680968771a4ec5ade6895b292df3 not found: ID does not exist" Oct 30 12:40:13.700390 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.696904 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm"] Oct 30 12:40:13.711675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:13.711643 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-q5glm"] Oct 30 12:40:15.091186 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:15.091150 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e88604-3549-4999-aefa-9bccc1042c87" path="/var/lib/kubelet/pods/87e88604-3549-4999-aefa-9bccc1042c87/volumes" Oct 30 12:40:21.453418 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:21.453380 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz"] Oct 30 12:40:21.453946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:21.453663 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" podUID="3b5da186-2a29-4bef-87fc-5dbfca607e5e" containerName="istio-proxy" containerID="cri-o://d968c5ce7853b15346671e690ad89043e4a465d02e8aa5c51688ec136dd5991c" gracePeriod=30 Oct 30 12:40:21.466630 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:21.466598 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2"] Oct 30 12:40:21.466852 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:21.466828 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" podUID="0f34a4f2-182f-48c8-9be1-442a72d04c71" containerName="istio-proxy" containerID="cri-o://23d17e3f7da9bf80562519af20ea7657d62907435c7ef179be837c1ab780918f" gracePeriod=30 Oct 30 12:40:21.471849 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:21.471814 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc"] Oct 30 12:40:21.472103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:21.472043 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" podUID="03176887-19c3-430c-8989-90572b562b75" containerName="istio-proxy" containerID="cri-o://0d1efabb70fea1f9a7792010cf958862032d04f738ea640c73dfb68918691d82" gracePeriod=30 Oct 30 12:40:23.599614 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:40:23.599575 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:40:26.721487 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.721450 4468 generic.go:356] "Generic (PLEG): container finished" podID="03176887-19c3-430c-8989-90572b562b75" containerID="0d1efabb70fea1f9a7792010cf958862032d04f738ea640c73dfb68918691d82" exitCode=0 Oct 30 12:40:26.721932 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.721577 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" event={"ID":"03176887-19c3-430c-8989-90572b562b75","Type":"ContainerDied","Data":"0d1efabb70fea1f9a7792010cf958862032d04f738ea640c73dfb68918691d82"} Oct 30 12:40:26.723592 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.723558 4468 generic.go:356] "Generic (PLEG): container finished" podID="3b5da186-2a29-4bef-87fc-5dbfca607e5e" containerID="d968c5ce7853b15346671e690ad89043e4a465d02e8aa5c51688ec136dd5991c" exitCode=0 Oct 30 12:40:26.723731 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.723684 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" event={"ID":"3b5da186-2a29-4bef-87fc-5dbfca607e5e","Type":"ContainerDied","Data":"d968c5ce7853b15346671e690ad89043e4a465d02e8aa5c51688ec136dd5991c"} Oct 30 12:40:26.725396 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.725337 4468 generic.go:356] "Generic (PLEG): container finished" podID="0f34a4f2-182f-48c8-9be1-442a72d04c71" containerID="23d17e3f7da9bf80562519af20ea7657d62907435c7ef179be837c1ab780918f" exitCode=0 Oct 30 12:40:26.725521 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.725411 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" event={"ID":"0f34a4f2-182f-48c8-9be1-442a72d04c71","Type":"ContainerDied","Data":"23d17e3f7da9bf80562519af20ea7657d62907435c7ef179be837c1ab780918f"} Oct 30 12:40:26.761951 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.761917 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:40:26.806466 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.806438 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:40:26.814767 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.814746 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:40:26.867178 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.867144 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-envoy\") pod \"0f34a4f2-182f-48c8-9be1-442a72d04c71\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " Oct 30 12:40:26.867339 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.867239 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-workload-socket\") pod \"0f34a4f2-182f-48c8-9be1-442a72d04c71\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " Oct 30 12:40:26.867390 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.867373 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-credential-socket\") pod \"0f34a4f2-182f-48c8-9be1-442a72d04c71\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.867478 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-podinfo\") pod \"0f34a4f2-182f-48c8-9be1-442a72d04c71\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.867925 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-credential-socket" pod "0f34a4f2-182f-48c8-9be1-442a72d04c71" (UID: "0f34a4f2-182f-48c8-9be1-442a72d04c71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.867944 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-data" pod "0f34a4f2-182f-48c8-9be1-442a72d04c71" (UID: "0f34a4f2-182f-48c8-9be1-442a72d04c71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.868009 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-data\") pod \"0f34a4f2-182f-48c8-9be1-442a72d04c71\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.868085 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5da186-2a29-4bef-87fc-5dbfca607e5e-kube-api-access-mkd85\" (UniqueName: \"kubernetes.io/projected/3b5da186-2a29-4bef-87fc-5dbfca607e5e-kube-api-access-mkd85\") pod \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.868143 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-workload-certs\") pod \"0f34a4f2-182f-48c8-9be1-442a72d04c71\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.868085 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-workload-socket" pod "0f34a4f2-182f-48c8-9be1-442a72d04c71" (UID: "0f34a4f2-182f-48c8-9be1-442a72d04c71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.868195 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f34a4f2-182f-48c8-9be1-442a72d04c71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f34a4f2-182f-48c8-9be1-442a72d04c71-istiod-ca-cert\") pod \"0f34a4f2-182f-48c8-9be1-442a72d04c71\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.868228 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-workload-socket\") pod \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.868260 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-token\" (UniqueName: \"kubernetes.io/projected/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-token\") pod \"0f34a4f2-182f-48c8-9be1-442a72d04c71\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.868315 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-data\") pod \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.868352 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03176887-19c3-430c-8989-90572b562b75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03176887-19c3-430c-8989-90572b562b75-istio-podinfo\") pod \"03176887-19c3-430c-8989-90572b562b75\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.868397 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-workload-socket\") pod \"03176887-19c3-430c-8989-90572b562b75\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.868442 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f34a4f2-182f-48c8-9be1-442a72d04c71-kube-api-access-829pw\" (UniqueName: \"kubernetes.io/projected/0f34a4f2-182f-48c8-9be1-442a72d04c71-kube-api-access-829pw\") pod \"0f34a4f2-182f-48c8-9be1-442a72d04c71\" (UID: \"0f34a4f2-182f-48c8-9be1-442a72d04c71\") " Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.868530 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-workload-certs" pod "0f34a4f2-182f-48c8-9be1-442a72d04c71" (UID: "0f34a4f2-182f-48c8-9be1-442a72d04c71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.868951 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-workload-socket" pod "3b5da186-2a29-4bef-87fc-5dbfca607e5e" (UID: "3b5da186-2a29-4bef-87fc-5dbfca607e5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.869028 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f34a4f2-182f-48c8-9be1-442a72d04c71-istiod-ca-cert" pod "0f34a4f2-182f-48c8-9be1-442a72d04c71" (UID: "0f34a4f2-182f-48c8-9be1-442a72d04c71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.869088 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.869110 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.869124 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.869137 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.867500 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.869151 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.870756 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.869386 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-workload-socket" pod "03176887-19c3-430c-8989-90572b562b75" (UID: "03176887-19c3-430c-8989-90572b562b75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:26.870756 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.869465 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-data" pod "3b5da186-2a29-4bef-87fc-5dbfca607e5e" (UID: "3b5da186-2a29-4bef-87fc-5dbfca607e5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:26.872877 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.872843 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f34a4f2-182f-48c8-9be1-442a72d04c71-kube-api-access-829pw" pod "0f34a4f2-182f-48c8-9be1-442a72d04c71" (UID: "0f34a4f2-182f-48c8-9be1-442a72d04c71"). InnerVolumeSpecName "kube-api-access-829pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:26.884551 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.879284 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-token" pod "0f34a4f2-182f-48c8-9be1-442a72d04c71" (UID: "0f34a4f2-182f-48c8-9be1-442a72d04c71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:26.884551 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.879329 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5da186-2a29-4bef-87fc-5dbfca607e5e-kube-api-access-mkd85" pod "3b5da186-2a29-4bef-87fc-5dbfca607e5e" (UID: "3b5da186-2a29-4bef-87fc-5dbfca607e5e"). InnerVolumeSpecName "kube-api-access-mkd85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:26.884551 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.879393 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-podinfo" pod "0f34a4f2-182f-48c8-9be1-442a72d04c71" (UID: "0f34a4f2-182f-48c8-9be1-442a72d04c71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:40:26.884551 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.880698 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-envoy" pod "0f34a4f2-182f-48c8-9be1-442a72d04c71" (UID: "0f34a4f2-182f-48c8-9be1-442a72d04c71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:26.884869 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.884707 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03176887-19c3-430c-8989-90572b562b75-istio-podinfo" pod "03176887-19c3-430c-8989-90572b562b75" (UID: "03176887-19c3-430c-8989-90572b562b75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:40:26.969753 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.969725 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-istio-data\") pod \"03176887-19c3-430c-8989-90572b562b75\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " Oct 30 12:40:26.969926 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.969766 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-istio-envoy\") pod \"03176887-19c3-430c-8989-90572b562b75\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " Oct 30 12:40:26.969926 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.969818 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-podinfo\") pod \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " Oct 30 12:40:26.969926 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.969839 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03176887-19c3-430c-8989-90572b562b75-istio-token\" (UniqueName: \"kubernetes.io/projected/03176887-19c3-430c-8989-90572b562b75-istio-token\") pod \"03176887-19c3-430c-8989-90572b562b75\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " Oct 30 12:40:26.969926 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.969855 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-workload-certs\") pod \"03176887-19c3-430c-8989-90572b562b75\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " Oct 30 12:40:26.969926 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.969883 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-workload-certs\") pod \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " Oct 30 12:40:26.969926 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.969911 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03176887-19c3-430c-8989-90572b562b75-kube-api-access-b5v7r\" (UniqueName: \"kubernetes.io/projected/03176887-19c3-430c-8989-90572b562b75-kube-api-access-b5v7r\") pod \"03176887-19c3-430c-8989-90572b562b75\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " Oct 30 12:40:26.970330 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.969965 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-credential-socket\") pod \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " Oct 30 12:40:26.970330 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.969990 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istiod-ca-cert\") pod \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " Oct 30 12:40:26.970330 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.970004 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-istio-data" pod "03176887-19c3-430c-8989-90572b562b75" (UID: "03176887-19c3-430c-8989-90572b562b75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:26.970330 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.970022 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-envoy\") pod \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " Oct 30 12:40:26.970330 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.970086 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03176887-19c3-430c-8989-90572b562b75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03176887-19c3-430c-8989-90572b562b75-istiod-ca-cert\") pod \"03176887-19c3-430c-8989-90572b562b75\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " Oct 30 12:40:26.970330 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.970121 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-token\" (UniqueName: \"kubernetes.io/projected/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-token\") pod \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\" (UID: \"3b5da186-2a29-4bef-87fc-5dbfca607e5e\") " Oct 30 12:40:26.970330 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.970166 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-credential-socket\") pod \"03176887-19c3-430c-8989-90572b562b75\" (UID: \"03176887-19c3-430c-8989-90572b562b75\") " Oct 30 12:40:26.970330 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.970177 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-workload-certs" pod "3b5da186-2a29-4bef-87fc-5dbfca607e5e" (UID: "3b5da186-2a29-4bef-87fc-5dbfca607e5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:26.970330 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.970249 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-workload-certs" pod "03176887-19c3-430c-8989-90572b562b75" (UID: "03176887-19c3-430c-8989-90572b562b75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:26.970330 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.970266 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-credential-socket" pod "3b5da186-2a29-4bef-87fc-5dbfca607e5e" (UID: "3b5da186-2a29-4bef-87fc-5dbfca607e5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:26.970867 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.970381 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-credential-socket" pod "03176887-19c3-430c-8989-90572b562b75" (UID: "03176887-19c3-430c-8989-90572b562b75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:26.970867 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.970689 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03176887-19c3-430c-8989-90572b562b75-istiod-ca-cert" pod "03176887-19c3-430c-8989-90572b562b75" (UID: "03176887-19c3-430c-8989-90572b562b75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:40:26.971587 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.971556 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.971696 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.971588 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.971696 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.971603 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.971696 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.971616 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.971696 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.971630 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03176887-19c3-430c-8989-90572b562b75-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.971696 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.971643 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.971696 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.971658 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.971696 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.971671 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkd85\" (UniqueName: \"kubernetes.io/projected/3b5da186-2a29-4bef-87fc-5dbfca607e5e-kube-api-access-mkd85\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.971696 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.971684 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f34a4f2-182f-48c8-9be1-442a72d04c71-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.971696 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.971697 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.972200 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.971711 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.972200 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.971723 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03176887-19c3-430c-8989-90572b562b75-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.972200 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.971735 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.972200 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.971747 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-829pw\" (UniqueName: \"kubernetes.io/projected/0f34a4f2-182f-48c8-9be1-442a72d04c71-kube-api-access-829pw\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.972200 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.971762 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f34a4f2-182f-48c8-9be1-442a72d04c71-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:26.972491 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.972468 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03176887-19c3-430c-8989-90572b562b75-istio-token" pod "03176887-19c3-430c-8989-90572b562b75" (UID: "03176887-19c3-430c-8989-90572b562b75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:26.972706 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.972678 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-istio-envoy" pod "03176887-19c3-430c-8989-90572b562b75" (UID: "03176887-19c3-430c-8989-90572b562b75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:26.972706 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.972692 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-envoy" pod "3b5da186-2a29-4bef-87fc-5dbfca607e5e" (UID: "3b5da186-2a29-4bef-87fc-5dbfca607e5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:26.972706 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.972685 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-podinfo" pod "3b5da186-2a29-4bef-87fc-5dbfca607e5e" (UID: "3b5da186-2a29-4bef-87fc-5dbfca607e5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:40:26.972928 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.972910 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03176887-19c3-430c-8989-90572b562b75-kube-api-access-b5v7r" pod "03176887-19c3-430c-8989-90572b562b75" (UID: "03176887-19c3-430c-8989-90572b562b75"). InnerVolumeSpecName "kube-api-access-b5v7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:26.973112 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.973057 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-token" pod "3b5da186-2a29-4bef-87fc-5dbfca607e5e" (UID: "3b5da186-2a29-4bef-87fc-5dbfca607e5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:26.973221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:26.973188 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istiod-ca-cert" pod "3b5da186-2a29-4bef-87fc-5dbfca607e5e" (UID: "3b5da186-2a29-4bef-87fc-5dbfca607e5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:40:27.072896 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.072812 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03176887-19c3-430c-8989-90572b562b75-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:27.072896 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.072842 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:27.072896 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.072851 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03176887-19c3-430c-8989-90572b562b75-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:27.072896 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.072860 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5v7r\" (UniqueName: \"kubernetes.io/projected/03176887-19c3-430c-8989-90572b562b75-kube-api-access-b5v7r\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:27.072896 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.072869 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:27.072896 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.072884 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:27.072896 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.072891 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5da186-2a29-4bef-87fc-5dbfca607e5e-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:27.733653 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.733617 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" Oct 30 12:40:27.734159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.733652 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc" event={"ID":"03176887-19c3-430c-8989-90572b562b75","Type":"ContainerDied","Data":"5834d84c9f0ed57ff5a9e9732bfb6b2e08efb37106af70ee25cf629154ea8c49"} Oct 30 12:40:27.734159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.733728 4468 scope.go:122] "RemoveContainer" containerID="0d1efabb70fea1f9a7792010cf958862032d04f738ea640c73dfb68918691d82" Oct 30 12:40:27.735195 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.735162 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" Oct 30 12:40:27.735286 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.735202 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz" event={"ID":"3b5da186-2a29-4bef-87fc-5dbfca607e5e","Type":"ContainerDied","Data":"4cc017ec76f541a5f7c9de26100a16e8c4669c1390254db87523214948161772"} Oct 30 12:40:27.736806 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.736784 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" event={"ID":"0f34a4f2-182f-48c8-9be1-442a72d04c71","Type":"ContainerDied","Data":"07dbb6ccbee1edcc3a90cf2b498ca349572e8c1a080b0423e96028519e56de4b"} Oct 30 12:40:27.736806 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.736804 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2" Oct 30 12:40:27.745414 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.745395 4468 scope.go:122] "RemoveContainer" containerID="d968c5ce7853b15346671e690ad89043e4a465d02e8aa5c51688ec136dd5991c" Oct 30 12:40:27.754862 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.754846 4468 scope.go:122] "RemoveContainer" containerID="23d17e3f7da9bf80562519af20ea7657d62907435c7ef179be837c1ab780918f" Oct 30 12:40:27.792444 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.792420 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2"] Oct 30 12:40:27.805903 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.805879 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-8f6h2"] Oct 30 12:40:27.843347 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.843325 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz"] Oct 30 12:40:27.847975 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.847951 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-lzlcz"] Oct 30 12:40:27.887198 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.887177 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc"] Oct 30 12:40:27.891448 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:27.891425 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4w4qc"] Oct 30 12:40:28.616714 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:28.616679 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67"] Oct 30 12:40:28.616939 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:28.616913 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" podUID="4fc449d1-bb52-4529-bb7c-407d5277050f" containerName="istio-proxy" containerID="cri-o://65ca75bfbcb6d215d07e515da0bb67faf8a5184f9841d68372e3fd5f1cc26d00" gracePeriod=30 Oct 30 12:40:29.092542 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:29.092508 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03176887-19c3-430c-8989-90572b562b75" path="/var/lib/kubelet/pods/03176887-19c3-430c-8989-90572b562b75/volumes" Oct 30 12:40:29.093043 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:29.093028 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f34a4f2-182f-48c8-9be1-442a72d04c71" path="/var/lib/kubelet/pods/0f34a4f2-182f-48c8-9be1-442a72d04c71/volumes" Oct 30 12:40:29.093494 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:29.093481 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b5da186-2a29-4bef-87fc-5dbfca607e5e" path="/var/lib/kubelet/pods/3b5da186-2a29-4bef-87fc-5dbfca607e5e/volumes" Oct 30 12:40:30.721868 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.721834 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw"] Oct 30 12:40:30.722517 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.722502 4468 cpu_manager.go:414] "containerMap: removing container" podUID="87e88604-3549-4999-aefa-9bccc1042c87" containerName="istio-proxy" Oct 30 12:40:30.722517 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.722517 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e88604-3549-4999-aefa-9bccc1042c87" containerName="istio-proxy" Oct 30 12:40:30.722593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.722533 4468 cpu_manager.go:414] "containerMap: removing container" podUID="0f34a4f2-182f-48c8-9be1-442a72d04c71" containerName="istio-proxy" Oct 30 12:40:30.722593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.722540 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f34a4f2-182f-48c8-9be1-442a72d04c71" containerName="istio-proxy" Oct 30 12:40:30.722593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.722555 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3b5da186-2a29-4bef-87fc-5dbfca607e5e" containerName="istio-proxy" Oct 30 12:40:30.722593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.722560 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5da186-2a29-4bef-87fc-5dbfca607e5e" containerName="istio-proxy" Oct 30 12:40:30.722593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.722574 4468 cpu_manager.go:414] "containerMap: removing container" podUID="03176887-19c3-430c-8989-90572b562b75" containerName="istio-proxy" Oct 30 12:40:30.722593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.722578 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03176887-19c3-430c-8989-90572b562b75" containerName="istio-proxy" Oct 30 12:40:30.722773 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.722684 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e88604-3549-4999-aefa-9bccc1042c87" containerName="istio-proxy" Oct 30 12:40:30.722773 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.722695 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f34a4f2-182f-48c8-9be1-442a72d04c71" containerName="istio-proxy" Oct 30 12:40:30.722773 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.722703 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b5da186-2a29-4bef-87fc-5dbfca607e5e" containerName="istio-proxy" Oct 30 12:40:30.722773 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.722710 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="03176887-19c3-430c-8989-90572b562b75" containerName="istio-proxy" Oct 30 12:40:30.728116 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.728099 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" Oct 30 12:40:30.746358 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.746328 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw"] Oct 30 12:40:30.753465 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.753441 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-dd0628-serving-cert\"" Oct 30 12:40:30.829240 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.829207 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b10586a-875c-4812-9bd2-0e65b81d0224-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw\" (UID: \"2b10586a-875c-4812-9bd2-0e65b81d0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" Oct 30 12:40:30.829438 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.829417 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b10586a-875c-4812-9bd2-0e65b81d0224-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw\" (UID: \"2b10586a-875c-4812-9bd2-0e65b81d0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" Oct 30 12:40:30.829491 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.829458 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b10586a-875c-4812-9bd2-0e65b81d0224-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw\" (UID: \"2b10586a-875c-4812-9bd2-0e65b81d0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" Oct 30 12:40:30.829553 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.829541 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv94k\" (UniqueName: \"kubernetes.io/projected/2b10586a-875c-4812-9bd2-0e65b81d0224-kube-api-access-sv94k\") pod \"maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw\" (UID: \"2b10586a-875c-4812-9bd2-0e65b81d0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" Oct 30 12:40:30.946758 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.946264 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b10586a-875c-4812-9bd2-0e65b81d0224-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw\" (UID: \"2b10586a-875c-4812-9bd2-0e65b81d0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" Oct 30 12:40:30.946758 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.946492 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b10586a-875c-4812-9bd2-0e65b81d0224-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw\" (UID: \"2b10586a-875c-4812-9bd2-0e65b81d0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" Oct 30 12:40:30.946758 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.946537 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b10586a-875c-4812-9bd2-0e65b81d0224-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw\" (UID: \"2b10586a-875c-4812-9bd2-0e65b81d0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" Oct 30 12:40:30.946758 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.946611 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv94k\" (UniqueName: \"kubernetes.io/projected/2b10586a-875c-4812-9bd2-0e65b81d0224-kube-api-access-sv94k\") pod \"maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw\" (UID: \"2b10586a-875c-4812-9bd2-0e65b81d0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" Oct 30 12:40:30.958109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.948095 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b10586a-875c-4812-9bd2-0e65b81d0224-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw\" (UID: \"2b10586a-875c-4812-9bd2-0e65b81d0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" Oct 30 12:40:30.958109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.957358 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b10586a-875c-4812-9bd2-0e65b81d0224-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw\" (UID: \"2b10586a-875c-4812-9bd2-0e65b81d0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" Oct 30 12:40:30.981139 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:30.980683 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b10586a-875c-4812-9bd2-0e65b81d0224-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw\" (UID: \"2b10586a-875c-4812-9bd2-0e65b81d0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" Oct 30 12:40:31.012090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:31.009796 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv94k\" (UniqueName: \"kubernetes.io/projected/2b10586a-875c-4812-9bd2-0e65b81d0224-kube-api-access-sv94k\") pod \"maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw\" (UID: \"2b10586a-875c-4812-9bd2-0e65b81d0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" Oct 30 12:40:31.043815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:31.043780 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" Oct 30 12:40:31.353480 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:40:31.353451 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b10586a_875c_4812_9bd2_0e65b81d0224.slice/crio-d583b58500fd08fab0afddde676e78c91545c571ce0ae07c2283140ac7d9fc19 WatchSource:0}: Error finding container d583b58500fd08fab0afddde676e78c91545c571ce0ae07c2283140ac7d9fc19: Status 404 returned error can't find the container with id d583b58500fd08fab0afddde676e78c91545c571ce0ae07c2283140ac7d9fc19 Oct 30 12:40:31.356393 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:31.356372 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw"] Oct 30 12:40:31.778296 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:31.778258 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" event={"ID":"2b10586a-875c-4812-9bd2-0e65b81d0224","Type":"ContainerStarted","Data":"d583b58500fd08fab0afddde676e78c91545c571ce0ae07c2283140ac7d9fc19"} Oct 30 12:40:32.785970 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:32.785930 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" event={"ID":"2b10586a-875c-4812-9bd2-0e65b81d0224","Type":"ContainerStarted","Data":"230d8431a796d2c44e22ead60a63c44dcabfa7b00db2d7c1ae4bda649be239e4"} Oct 30 12:40:32.786489 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:32.786114 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" Oct 30 12:40:32.853206 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:32.853153 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" podStartSLOduration=2.465582324 podStartE2EDuration="2.853138768s" totalImagesPullingTime="387.556444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:32.850683034 +0000 UTC m=+2783.884821432" watchObservedRunningTime="2026-10-30 12:40:32.853138768 +0000 UTC m=+2783.887277164" Oct 30 12:40:33.799645 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:33.799616 4468 generic.go:356] "Generic (PLEG): container finished" podID="4fc449d1-bb52-4529-bb7c-407d5277050f" containerID="65ca75bfbcb6d215d07e515da0bb67faf8a5184f9841d68372e3fd5f1cc26d00" exitCode=0 Oct 30 12:40:33.800637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:33.800605 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" event={"ID":"4fc449d1-bb52-4529-bb7c-407d5277050f","Type":"ContainerDied","Data":"65ca75bfbcb6d215d07e515da0bb67faf8a5184f9841d68372e3fd5f1cc26d00"} Oct 30 12:40:33.885309 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:33.885245 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:40:34.037768 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.037733 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-workload-certs\") pod \"4fc449d1-bb52-4529-bb7c-407d5277050f\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " Oct 30 12:40:34.037768 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.037774 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-podinfo\") pod \"4fc449d1-bb52-4529-bb7c-407d5277050f\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " Oct 30 12:40:34.037969 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.037812 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-data\") pod \"4fc449d1-bb52-4529-bb7c-407d5277050f\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " Oct 30 12:40:34.037969 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.037837 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-token\" (UniqueName: \"kubernetes.io/projected/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-token\") pod \"4fc449d1-bb52-4529-bb7c-407d5277050f\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " Oct 30 12:40:34.037969 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.037916 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-envoy\") pod \"4fc449d1-bb52-4529-bb7c-407d5277050f\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " Oct 30 12:40:34.037969 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.037964 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fc449d1-bb52-4529-bb7c-407d5277050f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc449d1-bb52-4529-bb7c-407d5277050f-istiod-ca-cert\") pod \"4fc449d1-bb52-4529-bb7c-407d5277050f\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " Oct 30 12:40:34.038247 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.038008 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-workload-socket\") pod \"4fc449d1-bb52-4529-bb7c-407d5277050f\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " Oct 30 12:40:34.038247 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.038044 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-credential-socket\") pod \"4fc449d1-bb52-4529-bb7c-407d5277050f\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " Oct 30 12:40:34.038247 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.038062 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-workload-certs" pod "4fc449d1-bb52-4529-bb7c-407d5277050f" (UID: "4fc449d1-bb52-4529-bb7c-407d5277050f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:34.038247 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.038230 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fc449d1-bb52-4529-bb7c-407d5277050f-kube-api-access-rmc5h\" (UniqueName: \"kubernetes.io/projected/4fc449d1-bb52-4529-bb7c-407d5277050f-kube-api-access-rmc5h\") pod \"4fc449d1-bb52-4529-bb7c-407d5277050f\" (UID: \"4fc449d1-bb52-4529-bb7c-407d5277050f\") " Oct 30 12:40:34.038464 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.038393 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-data" pod "4fc449d1-bb52-4529-bb7c-407d5277050f" (UID: "4fc449d1-bb52-4529-bb7c-407d5277050f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:34.038464 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.038407 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-workload-socket" pod "4fc449d1-bb52-4529-bb7c-407d5277050f" (UID: "4fc449d1-bb52-4529-bb7c-407d5277050f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:34.038633 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.038606 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-credential-socket" pod "4fc449d1-bb52-4529-bb7c-407d5277050f" (UID: "4fc449d1-bb52-4529-bb7c-407d5277050f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:34.038633 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.038518 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc449d1-bb52-4529-bb7c-407d5277050f-istiod-ca-cert" pod "4fc449d1-bb52-4529-bb7c-407d5277050f" (UID: "4fc449d1-bb52-4529-bb7c-407d5277050f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:40:34.038976 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.038951 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:34.039056 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.038980 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:34.039056 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.038994 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc449d1-bb52-4529-bb7c-407d5277050f-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:34.039056 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.039007 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:34.039056 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.039020 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:34.040386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.040347 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-token" pod "4fc449d1-bb52-4529-bb7c-407d5277050f" (UID: "4fc449d1-bb52-4529-bb7c-407d5277050f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:34.040492 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.040424 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-podinfo" pod "4fc449d1-bb52-4529-bb7c-407d5277050f" (UID: "4fc449d1-bb52-4529-bb7c-407d5277050f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:40:34.040580 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.040500 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc449d1-bb52-4529-bb7c-407d5277050f-kube-api-access-rmc5h" pod "4fc449d1-bb52-4529-bb7c-407d5277050f" (UID: "4fc449d1-bb52-4529-bb7c-407d5277050f"). InnerVolumeSpecName "kube-api-access-rmc5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:34.040661 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.040629 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-envoy" pod "4fc449d1-bb52-4529-bb7c-407d5277050f" (UID: "4fc449d1-bb52-4529-bb7c-407d5277050f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:34.140845 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.140765 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:34.140845 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.140794 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:34.140845 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.140803 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc449d1-bb52-4529-bb7c-407d5277050f-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:34.140845 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.140812 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmc5h\" (UniqueName: \"kubernetes.io/projected/4fc449d1-bb52-4529-bb7c-407d5277050f-kube-api-access-rmc5h\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:40:34.808218 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.808191 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" Oct 30 12:40:34.808700 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.808221 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67" event={"ID":"4fc449d1-bb52-4529-bb7c-407d5277050f","Type":"ContainerDied","Data":"a246717f7ade0b11b73999be6eefd701b75814b7d3e62130f8995a70a1a81cfa"} Oct 30 12:40:34.808700 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.808294 4468 scope.go:122] "RemoveContainer" containerID="65ca75bfbcb6d215d07e515da0bb67faf8a5184f9841d68372e3fd5f1cc26d00" Oct 30 12:40:34.881725 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.881692 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67"] Oct 30 12:40:34.895191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:34.895162 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-zsp67"] Oct 30 12:40:35.090563 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:35.090491 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fc449d1-bb52-4529-bb7c-407d5277050f" path="/var/lib/kubelet/pods/4fc449d1-bb52-4529-bb7c-407d5277050f/volumes" Oct 30 12:40:38.806213 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:38.806187 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" Oct 30 12:40:47.516898 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:47.516855 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw"] Oct 30 12:40:53.602977 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:40:53.602926 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:40:56.941824 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:56.941781 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11587-c10d24"] Oct 30 12:40:56.942802 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:56.942767 4468 cpu_manager.go:414] "containerMap: removing container" podUID="4fc449d1-bb52-4529-bb7c-407d5277050f" containerName="istio-proxy" Oct 30 12:40:56.942802 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:56.942802 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fc449d1-bb52-4529-bb7c-407d5277050f" containerName="istio-proxy" Oct 30 12:40:56.943006 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:56.942983 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fc449d1-bb52-4529-bb7c-407d5277050f" containerName="istio-proxy" Oct 30 12:40:56.947769 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:56.947735 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11587-c10d24" Oct 30 12:40:56.987028 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:56.986992 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ttfhc\"" Oct 30 12:40:56.995750 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:56.995708 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11587-c10d24"] Oct 30 12:40:56.999752 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:56.999718 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkb9v\" (UniqueName: \"kubernetes.io/projected/95870260-41ea-438b-b617-b9bb3ab8c623-kube-api-access-wkb9v\") pod \"test-curl-11587-c10d24\" (UID: \"95870260-41ea-438b-b617-b9bb3ab8c623\") " pod="opendatahub/test-curl-11587-c10d24" Oct 30 12:40:57.101534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:57.101483 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkb9v\" (UniqueName: \"kubernetes.io/projected/95870260-41ea-438b-b617-b9bb3ab8c623-kube-api-access-wkb9v\") pod \"test-curl-11587-c10d24\" (UID: \"95870260-41ea-438b-b617-b9bb3ab8c623\") " pod="opendatahub/test-curl-11587-c10d24" Oct 30 12:40:57.126839 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:57.126806 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkb9v\" (UniqueName: \"kubernetes.io/projected/95870260-41ea-438b-b617-b9bb3ab8c623-kube-api-access-wkb9v\") pod \"test-curl-11587-c10d24\" (UID: \"95870260-41ea-438b-b617-b9bb3ab8c623\") " pod="opendatahub/test-curl-11587-c10d24" Oct 30 12:40:57.271967 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:57.271877 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11587-c10d24" Oct 30 12:40:57.583345 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:40:57.583292 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95870260_41ea_438b_b617_b9bb3ab8c623.slice/crio-4f3fce18b8bec0d187740629da057901b3df91a8db38672c2e1f530d31e0ee44 WatchSource:0}: Error finding container 4f3fce18b8bec0d187740629da057901b3df91a8db38672c2e1f530d31e0ee44: Status 404 returned error can't find the container with id 4f3fce18b8bec0d187740629da057901b3df91a8db38672c2e1f530d31e0ee44 Oct 30 12:40:57.585557 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:57.585536 4468 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:40:57.587406 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:57.587381 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11587-c10d24"] Oct 30 12:40:58.135859 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:40:58.135817 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11587-c10d24" event={"ID":"95870260-41ea-438b-b617-b9bb3ab8c623","Type":"ContainerStarted","Data":"4f3fce18b8bec0d187740629da057901b3df91a8db38672c2e1f530d31e0ee44"} Oct 30 12:41:00.211156 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:00.211116 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11587-c10d24" event={"ID":"95870260-41ea-438b-b617-b9bb3ab8c623","Type":"ContainerStarted","Data":"9895c5d16a9bed0b41f4698100a4717df932b2b4fe157626dc18580f38dbb534"} Oct 30 12:41:00.253181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:00.253107 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11587-c10d24" podStartSLOduration=2.135453763 podStartE2EDuration="4.253091013s" totalImagesPullingTime="2.11763725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:00.251020044 +0000 UTC m=+2811.285158440" watchObservedRunningTime="2026-10-30 12:41:00.253091013 +0000 UTC m=+2811.287229403" Oct 30 12:41:00.283873 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:00.283824 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8"] Oct 30 12:41:00.284062 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:00.284041 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" podUID="9d42ea5c-ec5b-433f-887a-76d2b8a68f46" containerName="istio-proxy" containerID="cri-o://cf334a47c91ac785d5e84ae7473aae0363d9deae0005b7a90fea4abf2bd946dd" gracePeriod=30 Oct 30 12:41:01.263527 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:01.263490 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11587-c10d24"] Oct 30 12:41:01.269125 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:01.269083 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11587-c10d24"] Oct 30 12:41:02.224561 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:02.224513 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11587-c10d24" podUID="95870260-41ea-438b-b617-b9bb3ab8c623" containerName="test-curl-11587-c10d24" containerID="cri-o://9895c5d16a9bed0b41f4698100a4717df932b2b4fe157626dc18580f38dbb534" gracePeriod=2 Oct 30 12:41:02.239571 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:41:02.239523 4468 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11587-c10d24\" is forbidden: User \"system:node:ip-10-0-131-76.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-76.ec2.internal' and this object" podUID="95870260-41ea-438b-b617-b9bb3ab8c623" pod="opendatahub/test-curl-11587-c10d24" Oct 30 12:41:02.538189 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:02.538062 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" podUID="1c00f519-078c-4b00-bd7f-a758e916a650" containerName="main" containerID="cri-o://6773647367c2da889f45659d8b5475439e5a42e8f567aaa81482fb2a6c2c3e33" gracePeriod=45 Oct 30 12:41:02.829662 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:02.829631 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:41:02.889545 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:41:02.889500 4468 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11587-c10d24\" is forbidden: User \"system:node:ip-10-0-131-76.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-76.ec2.internal' and this object" podUID="95870260-41ea-438b-b617-b9bb3ab8c623" pod="opendatahub/test-curl-11587-c10d24" Oct 30 12:41:02.921183 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:02.921150 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-model-cache\") pod \"1c00f519-078c-4b00-bd7f-a758e916a650\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " Oct 30 12:41:02.921419 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:02.921310 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c00f519-078c-4b00-bd7f-a758e916a650-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c00f519-078c-4b00-bd7f-a758e916a650-tls-certs\") pod \"1c00f519-078c-4b00-bd7f-a758e916a650\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " Oct 30 12:41:02.921419 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:02.921362 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-tmp-dir\") pod \"1c00f519-078c-4b00-bd7f-a758e916a650\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " Oct 30 12:41:02.921546 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:02.921468 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-home\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-home\") pod \"1c00f519-078c-4b00-bd7f-a758e916a650\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " Oct 30 12:41:02.921609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:02.921556 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c00f519-078c-4b00-bd7f-a758e916a650-kube-api-access-ffsnn\" (UniqueName: \"kubernetes.io/projected/1c00f519-078c-4b00-bd7f-a758e916a650-kube-api-access-ffsnn\") pod \"1c00f519-078c-4b00-bd7f-a758e916a650\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " Oct 30 12:41:02.921689 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:02.921667 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-dshm\") pod \"1c00f519-078c-4b00-bd7f-a758e916a650\" (UID: \"1c00f519-078c-4b00-bd7f-a758e916a650\") " Oct 30 12:41:02.923452 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:02.923420 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-home" pod "1c00f519-078c-4b00-bd7f-a758e916a650" (UID: "1c00f519-078c-4b00-bd7f-a758e916a650"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:02.923879 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:02.923855 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-model-cache" pod "1c00f519-078c-4b00-bd7f-a758e916a650" (UID: "1c00f519-078c-4b00-bd7f-a758e916a650"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:02.924294 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:02.924272 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-tmp-dir" pod "1c00f519-078c-4b00-bd7f-a758e916a650" (UID: "1c00f519-078c-4b00-bd7f-a758e916a650"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:02.936608 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:02.936559 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-dshm" pod "1c00f519-078c-4b00-bd7f-a758e916a650" (UID: "1c00f519-078c-4b00-bd7f-a758e916a650"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:02.936870 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:02.936612 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c00f519-078c-4b00-bd7f-a758e916a650-kube-api-access-ffsnn" pod "1c00f519-078c-4b00-bd7f-a758e916a650" (UID: "1c00f519-078c-4b00-bd7f-a758e916a650"). InnerVolumeSpecName "kube-api-access-ffsnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:02.938219 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:02.938162 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c00f519-078c-4b00-bd7f-a758e916a650-tls-certs" pod "1c00f519-078c-4b00-bd7f-a758e916a650" (UID: "1c00f519-078c-4b00-bd7f-a758e916a650"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:41:03.051158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:03.028166 4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-model-cache\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:03.051158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:03.033205 4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c00f519-078c-4b00-bd7f-a758e916a650-tls-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:03.051158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:03.033239 4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-tmp-dir\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:03.051158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:03.033252 4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-home\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:03.051158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:03.033267 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffsnn\" (UniqueName: \"kubernetes.io/projected/1c00f519-078c-4b00-bd7f-a758e916a650-kube-api-access-ffsnn\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:03.051158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:03.033282 4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c00f519-078c-4b00-bd7f-a758e916a650-dshm\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:03.237171 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:03.237132 4468 generic.go:356] "Generic (PLEG): container finished" podID="1c00f519-078c-4b00-bd7f-a758e916a650" containerID="6773647367c2da889f45659d8b5475439e5a42e8f567aaa81482fb2a6c2c3e33" exitCode=0 Oct 30 12:41:03.237412 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:03.237190 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" Oct 30 12:41:03.237412 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:03.237235 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" event={"ID":"1c00f519-078c-4b00-bd7f-a758e916a650","Type":"ContainerDied","Data":"6773647367c2da889f45659d8b5475439e5a42e8f567aaa81482fb2a6c2c3e33"} Oct 30 12:41:03.237412 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:03.237292 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw" event={"ID":"1c00f519-078c-4b00-bd7f-a758e916a650","Type":"ContainerDied","Data":"4c212be1cd695812b1a19064bf9603de03b6699eb25539946eace4268dbebdc2"} Oct 30 12:41:03.237412 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:03.237316 4468 scope.go:122] "RemoveContainer" containerID="6773647367c2da889f45659d8b5475439e5a42e8f567aaa81482fb2a6c2c3e33" Oct 30 12:41:03.249774 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:03.249749 4468 scope.go:122] "RemoveContainer" containerID="6773647367c2da889f45659d8b5475439e5a42e8f567aaa81482fb2a6c2c3e33" Oct 30 12:41:03.250145 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:41:03.250123 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6773647367c2da889f45659d8b5475439e5a42e8f567aaa81482fb2a6c2c3e33\": container with ID starting with 6773647367c2da889f45659d8b5475439e5a42e8f567aaa81482fb2a6c2c3e33 not found: ID does not exist" containerID="6773647367c2da889f45659d8b5475439e5a42e8f567aaa81482fb2a6c2c3e33" Oct 30 12:41:03.250229 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:03.250153 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6773647367c2da889f45659d8b5475439e5a42e8f567aaa81482fb2a6c2c3e33"} err="failed to get container status \"6773647367c2da889f45659d8b5475439e5a42e8f567aaa81482fb2a6c2c3e33\": rpc error: code = NotFound desc = could not find container \"6773647367c2da889f45659d8b5475439e5a42e8f567aaa81482fb2a6c2c3e33\": container with ID starting with 6773647367c2da889f45659d8b5475439e5a42e8f567aaa81482fb2a6c2c3e33 not found: ID does not exist" Oct 30 12:41:03.296198 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:03.296158 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw"] Oct 30 12:41:03.300868 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:03.300839 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-42220c/auth-test-model-42220c-kserve-7667b48d65-9srkw"] Oct 30 12:41:04.456380 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:04.456349 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11587-c10d24" Oct 30 12:41:04.483498 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:04.483401 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:41:04.483498 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:04.483485 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" podUID="9d42ea5c-ec5b-433f-887a-76d2b8a68f46" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:41:04.556837 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:04.556799 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95870260-41ea-438b-b617-b9bb3ab8c623-kube-api-access-wkb9v\" (UniqueName: \"kubernetes.io/projected/95870260-41ea-438b-b617-b9bb3ab8c623-kube-api-access-wkb9v\") pod \"95870260-41ea-438b-b617-b9bb3ab8c623\" (UID: \"95870260-41ea-438b-b617-b9bb3ab8c623\") " Oct 30 12:41:04.559213 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:04.559174 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95870260-41ea-438b-b617-b9bb3ab8c623-kube-api-access-wkb9v" pod "95870260-41ea-438b-b617-b9bb3ab8c623" (UID: "95870260-41ea-438b-b617-b9bb3ab8c623"). InnerVolumeSpecName "kube-api-access-wkb9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:04.658457 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:04.658424 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkb9v\" (UniqueName: \"kubernetes.io/projected/95870260-41ea-438b-b617-b9bb3ab8c623-kube-api-access-wkb9v\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:05.091393 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.091354 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c00f519-078c-4b00-bd7f-a758e916a650" path="/var/lib/kubelet/pods/1c00f519-078c-4b00-bd7f-a758e916a650/volumes" Oct 30 12:41:05.091892 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.091872 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95870260-41ea-438b-b617-b9bb3ab8c623" path="/var/lib/kubelet/pods/95870260-41ea-438b-b617-b9bb3ab8c623/volumes" Oct 30 12:41:05.310943 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.309600 4468 generic.go:356] "Generic (PLEG): container finished" podID="95870260-41ea-438b-b617-b9bb3ab8c623" containerID="9895c5d16a9bed0b41f4698100a4717df932b2b4fe157626dc18580f38dbb534" exitCode=137 Oct 30 12:41:05.310943 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.309736 4468 scope.go:122] "RemoveContainer" containerID="9895c5d16a9bed0b41f4698100a4717df932b2b4fe157626dc18580f38dbb534" Oct 30 12:41:05.310943 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.309870 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11587-c10d24" Oct 30 12:41:05.404375 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.404347 4468 scope.go:122] "RemoveContainer" containerID="9895c5d16a9bed0b41f4698100a4717df932b2b4fe157626dc18580f38dbb534" Oct 30 12:41:05.404848 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:41:05.404819 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9895c5d16a9bed0b41f4698100a4717df932b2b4fe157626dc18580f38dbb534\": container with ID starting with 9895c5d16a9bed0b41f4698100a4717df932b2b4fe157626dc18580f38dbb534 not found: ID does not exist" containerID="9895c5d16a9bed0b41f4698100a4717df932b2b4fe157626dc18580f38dbb534" Oct 30 12:41:05.404942 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.404862 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9895c5d16a9bed0b41f4698100a4717df932b2b4fe157626dc18580f38dbb534"} err="failed to get container status \"9895c5d16a9bed0b41f4698100a4717df932b2b4fe157626dc18580f38dbb534\": rpc error: code = NotFound desc = could not find container \"9895c5d16a9bed0b41f4698100a4717df932b2b4fe157626dc18580f38dbb534\": container with ID starting with 9895c5d16a9bed0b41f4698100a4717df932b2b4fe157626dc18580f38dbb534 not found: ID does not exist" Oct 30 12:41:05.425591 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.425538 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf"] Oct 30 12:41:05.426271 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.426253 4468 cpu_manager.go:414] "containerMap: removing container" podUID="1c00f519-078c-4b00-bd7f-a758e916a650" containerName="main" Oct 30 12:41:05.426343 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.426273 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c00f519-078c-4b00-bd7f-a758e916a650" containerName="main" Oct 30 12:41:05.426343 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.426297 4468 cpu_manager.go:414] "containerMap: removing container" podUID="95870260-41ea-438b-b617-b9bb3ab8c623" containerName="test-curl-11587-c10d24" Oct 30 12:41:05.426343 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.426304 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95870260-41ea-438b-b617-b9bb3ab8c623" containerName="test-curl-11587-c10d24" Oct 30 12:41:05.426444 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.426419 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c00f519-078c-4b00-bd7f-a758e916a650" containerName="main" Oct 30 12:41:05.426444 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.426428 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="95870260-41ea-438b-b617-b9bb3ab8c623" containerName="test-curl-11587-c10d24" Oct 30 12:41:05.467081 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.449651 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.481307 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.481268 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-6ad1da-openshift-default-dockercfg-hrjjc\"" Oct 30 12:41:05.495862 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.495714 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf"] Oct 30 12:41:05.655424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.655322 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db33d91a-07fb-4c52-832e-0440bc4654af-istiod-ca-cert\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.655700 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.655676 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-istio-data\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.655894 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.655877 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db33d91a-07fb-4c52-832e-0440bc4654af-istio-token\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.656060 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.656041 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-workload-certs\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.656265 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.656248 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-workload-socket\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.656448 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.656434 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-istio-envoy\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.656599 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.656583 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr9mt\" (UniqueName: \"kubernetes.io/projected/db33d91a-07fb-4c52-832e-0440bc4654af-kube-api-access-vr9mt\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.656746 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.656732 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-credential-socket\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.656926 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.656911 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db33d91a-07fb-4c52-832e-0440bc4654af-istio-podinfo\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.758806 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.758740 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-workload-certs\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.759021 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.758835 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-workload-socket\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.759021 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.758959 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-istio-envoy\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.759184 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.759040 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr9mt\" (UniqueName: \"kubernetes.io/projected/db33d91a-07fb-4c52-832e-0440bc4654af-kube-api-access-vr9mt\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.759184 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.759135 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-credential-socket\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.760353 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.760183 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db33d91a-07fb-4c52-832e-0440bc4654af-istio-podinfo\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.760519 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.760428 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db33d91a-07fb-4c52-832e-0440bc4654af-istiod-ca-cert\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.760519 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.760468 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-istio-data\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.760644 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.760604 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db33d91a-07fb-4c52-832e-0440bc4654af-istio-token\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.762113 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.762001 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-workload-certs\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.762113 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.762017 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-credential-socket\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.762393 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.762371 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-workload-socket\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.762634 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.762599 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-istio-data\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.763150 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.763126 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db33d91a-07fb-4c52-832e-0440bc4654af-istiod-ca-cert\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.765186 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.765039 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-istio-envoy\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.765616 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.765593 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db33d91a-07fb-4c52-832e-0440bc4654af-istio-podinfo\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.785985 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.785945 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db33d91a-07fb-4c52-832e-0440bc4654af-istio-token\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.786294 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.786275 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr9mt\" (UniqueName: \"kubernetes.io/projected/db33d91a-07fb-4c52-832e-0440bc4654af-kube-api-access-vr9mt\") pod \"e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.808629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.808589 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:05.824576 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.824547 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:41:05.862734 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.862697 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-envoy\") pod \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " Oct 30 12:41:05.862917 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.862812 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istiod-ca-cert\") pod \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " Oct 30 12:41:05.862917 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.862855 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-credential-socket\") pod \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " Oct 30 12:41:05.863039 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.862933 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-workload-certs\") pod \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " Oct 30 12:41:05.863039 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.862971 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-workload-socket\") pod \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " Oct 30 12:41:05.863155 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.863053 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-podinfo\") pod \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " Oct 30 12:41:05.863155 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.863105 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-kube-api-access-9qtwg\" (UniqueName: \"kubernetes.io/projected/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-kube-api-access-9qtwg\") pod \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " Oct 30 12:41:05.863277 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.863241 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-token\" (UniqueName: \"kubernetes.io/projected/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-token\") pod \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " Oct 30 12:41:05.863339 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.863273 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-workload-certs" pod "9d42ea5c-ec5b-433f-887a-76d2b8a68f46" (UID: "9d42ea5c-ec5b-433f-887a-76d2b8a68f46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:05.863339 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.863283 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-workload-socket" pod "9d42ea5c-ec5b-433f-887a-76d2b8a68f46" (UID: "9d42ea5c-ec5b-433f-887a-76d2b8a68f46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:05.863499 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.863377 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istiod-ca-cert" pod "9d42ea5c-ec5b-433f-887a-76d2b8a68f46" (UID: "9d42ea5c-ec5b-433f-887a-76d2b8a68f46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:41:05.863628 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.863598 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-credential-socket" pod "9d42ea5c-ec5b-433f-887a-76d2b8a68f46" (UID: "9d42ea5c-ec5b-433f-887a-76d2b8a68f46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:05.863628 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.863308 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-data\") pod \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\" (UID: \"9d42ea5c-ec5b-433f-887a-76d2b8a68f46\") " Oct 30 12:41:05.864355 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.864328 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:05.864438 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.864361 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:05.864438 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.864376 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:05.864438 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.864388 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:05.864910 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.864863 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-data" pod "9d42ea5c-ec5b-433f-887a-76d2b8a68f46" (UID: "9d42ea5c-ec5b-433f-887a-76d2b8a68f46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:05.865919 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.865891 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-envoy" pod "9d42ea5c-ec5b-433f-887a-76d2b8a68f46" (UID: "9d42ea5c-ec5b-433f-887a-76d2b8a68f46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:05.866107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.866050 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-podinfo" pod "9d42ea5c-ec5b-433f-887a-76d2b8a68f46" (UID: "9d42ea5c-ec5b-433f-887a-76d2b8a68f46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:41:05.866780 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.866748 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-token" pod "9d42ea5c-ec5b-433f-887a-76d2b8a68f46" (UID: "9d42ea5c-ec5b-433f-887a-76d2b8a68f46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:05.866943 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.866915 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-kube-api-access-9qtwg" pod "9d42ea5c-ec5b-433f-887a-76d2b8a68f46" (UID: "9d42ea5c-ec5b-433f-887a-76d2b8a68f46"). InnerVolumeSpecName "kube-api-access-9qtwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:05.966152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.966111 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:05.966152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.966155 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qtwg\" (UniqueName: \"kubernetes.io/projected/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-kube-api-access-9qtwg\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:05.966520 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.966172 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:05.966520 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.966185 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:05.966520 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:05.966197 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d42ea5c-ec5b-433f-887a-76d2b8a68f46-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:06.142177 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.142137 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22"] Oct 30 12:41:06.142877 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.142857 4468 cpu_manager.go:414] "containerMap: removing container" podUID="9d42ea5c-ec5b-433f-887a-76d2b8a68f46" containerName="istio-proxy" Oct 30 12:41:06.142877 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.142876 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d42ea5c-ec5b-433f-887a-76d2b8a68f46" containerName="istio-proxy" Oct 30 12:41:06.143120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.142993 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d42ea5c-ec5b-433f-887a-76d2b8a68f46" containerName="istio-proxy" Oct 30 12:41:06.148016 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.147986 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" Oct 30 12:41:06.172241 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.172211 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22"] Oct 30 12:41:06.173096 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.173053 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dc4l\" (UniqueName: \"kubernetes.io/projected/56adb9ab-1a6f-497c-8987-33123b8eb06a-kube-api-access-2dc4l\") pod \"payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22\" (UID: \"56adb9ab-1a6f-497c-8987-33123b8eb06a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" Oct 30 12:41:06.173712 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.173676 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56adb9ab-1a6f-497c-8987-33123b8eb06a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22\" (UID: \"56adb9ab-1a6f-497c-8987-33123b8eb06a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" Oct 30 12:41:06.174329 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.174298 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a786af\"" Oct 30 12:41:06.183390 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.183271 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf"] Oct 30 12:41:06.185778 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.185547 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:06.185778 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.185652 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:06.185778 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.185697 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:06.223630 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.223499 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d"] Oct 30 12:41:06.229841 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.229813 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" Oct 30 12:41:06.246224 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.246193 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d"] Oct 30 12:41:06.247032 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.246617 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a786af-dockercfg-5rwwm\"" Oct 30 12:41:06.280412 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.280347 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56adb9ab-1a6f-497c-8987-33123b8eb06a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22\" (UID: \"56adb9ab-1a6f-497c-8987-33123b8eb06a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" Oct 30 12:41:06.280676 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.280612 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dc4l\" (UniqueName: \"kubernetes.io/projected/56adb9ab-1a6f-497c-8987-33123b8eb06a-kube-api-access-2dc4l\") pod \"payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22\" (UID: \"56adb9ab-1a6f-497c-8987-33123b8eb06a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" Oct 30 12:41:06.283912 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.283865 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56adb9ab-1a6f-497c-8987-33123b8eb06a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22\" (UID: \"56adb9ab-1a6f-497c-8987-33123b8eb06a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" Oct 30 12:41:06.328536 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.324775 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dc4l\" (UniqueName: \"kubernetes.io/projected/56adb9ab-1a6f-497c-8987-33123b8eb06a-kube-api-access-2dc4l\") pod \"payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22\" (UID: \"56adb9ab-1a6f-497c-8987-33123b8eb06a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" Oct 30 12:41:06.378547 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.378506 4468 generic.go:356] "Generic (PLEG): container finished" podID="9d42ea5c-ec5b-433f-887a-76d2b8a68f46" containerID="cf334a47c91ac785d5e84ae7473aae0363d9deae0005b7a90fea4abf2bd946dd" exitCode=0 Oct 30 12:41:06.386364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.378761 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" event={"ID":"9d42ea5c-ec5b-433f-887a-76d2b8a68f46","Type":"ContainerDied","Data":"cf334a47c91ac785d5e84ae7473aae0363d9deae0005b7a90fea4abf2bd946dd"} Oct 30 12:41:06.386364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.378907 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" event={"ID":"9d42ea5c-ec5b-433f-887a-76d2b8a68f46","Type":"ContainerDied","Data":"5dcd1bcebbc10fad21c94d54c7515781bd7a82c7c0cca3b6f415de6fa25548db"} Oct 30 12:41:06.386364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.378988 4468 scope.go:122] "RemoveContainer" containerID="cf334a47c91ac785d5e84ae7473aae0363d9deae0005b7a90fea4abf2bd946dd" Oct 30 12:41:06.386364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.379320 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8" Oct 30 12:41:06.387766 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.387729 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64964ff4-80ce-4c89-af7b-19d3b24b07f4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d\" (UID: \"64964ff4-80ce-4c89-af7b-19d3b24b07f4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" Oct 30 12:41:06.389467 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.387907 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpcrd\" (UniqueName: \"kubernetes.io/projected/64964ff4-80ce-4c89-af7b-19d3b24b07f4-kube-api-access-xpcrd\") pod \"payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d\" (UID: \"64964ff4-80ce-4c89-af7b-19d3b24b07f4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" Oct 30 12:41:06.403549 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.403511 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" event={"ID":"db33d91a-07fb-4c52-832e-0440bc4654af","Type":"ContainerStarted","Data":"d07ce4b56f8827b7cdce31e95bca80a93cc343b2fce1a423e0e660c2e122c9cf"} Oct 30 12:41:06.416605 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.416548 4468 scope.go:122] "RemoveContainer" containerID="cf334a47c91ac785d5e84ae7473aae0363d9deae0005b7a90fea4abf2bd946dd" Oct 30 12:41:06.418500 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:41:06.418377 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf334a47c91ac785d5e84ae7473aae0363d9deae0005b7a90fea4abf2bd946dd\": container with ID starting with cf334a47c91ac785d5e84ae7473aae0363d9deae0005b7a90fea4abf2bd946dd not found: ID does not exist" containerID="cf334a47c91ac785d5e84ae7473aae0363d9deae0005b7a90fea4abf2bd946dd" Oct 30 12:41:06.418704 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.418517 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf334a47c91ac785d5e84ae7473aae0363d9deae0005b7a90fea4abf2bd946dd"} err="failed to get container status \"cf334a47c91ac785d5e84ae7473aae0363d9deae0005b7a90fea4abf2bd946dd\": rpc error: code = NotFound desc = could not find container \"cf334a47c91ac785d5e84ae7473aae0363d9deae0005b7a90fea4abf2bd946dd\": container with ID starting with cf334a47c91ac785d5e84ae7473aae0363d9deae0005b7a90fea4abf2bd946dd not found: ID does not exist" Oct 30 12:41:06.457911 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.457836 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" podStartSLOduration=1.457811379 podStartE2EDuration="1.457811379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:06.451595573 +0000 UTC m=+2817.485733970" watchObservedRunningTime="2026-10-30 12:41:06.457811379 +0000 UTC m=+2817.491949775" Oct 30 12:41:06.469665 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.469604 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" Oct 30 12:41:06.489382 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.489337 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64964ff4-80ce-4c89-af7b-19d3b24b07f4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d\" (UID: \"64964ff4-80ce-4c89-af7b-19d3b24b07f4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" Oct 30 12:41:06.489548 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.489517 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpcrd\" (UniqueName: \"kubernetes.io/projected/64964ff4-80ce-4c89-af7b-19d3b24b07f4-kube-api-access-xpcrd\") pod \"payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d\" (UID: \"64964ff4-80ce-4c89-af7b-19d3b24b07f4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" Oct 30 12:41:06.491120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.491038 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64964ff4-80ce-4c89-af7b-19d3b24b07f4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d\" (UID: \"64964ff4-80ce-4c89-af7b-19d3b24b07f4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" Oct 30 12:41:06.496282 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.496248 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8"] Oct 30 12:41:06.501490 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.501411 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rhhh8"] Oct 30 12:41:06.516025 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.515908 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpcrd\" (UniqueName: \"kubernetes.io/projected/64964ff4-80ce-4c89-af7b-19d3b24b07f4-kube-api-access-xpcrd\") pod \"payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d\" (UID: \"64964ff4-80ce-4c89-af7b-19d3b24b07f4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" Oct 30 12:41:06.633138 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.632835 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" Oct 30 12:41:06.807404 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.807358 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22"] Oct 30 12:41:06.809176 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.809101 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:06.820713 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:41:06.820661 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56adb9ab_1a6f_497c_8987_33123b8eb06a.slice/crio-4c717a82462a50efaf7f90cbf0f5c57dd9f6caea664128875723fb625f3c996d WatchSource:0}: Error finding container 4c717a82462a50efaf7f90cbf0f5c57dd9f6caea664128875723fb625f3c996d: Status 404 returned error can't find the container with id 4c717a82462a50efaf7f90cbf0f5c57dd9f6caea664128875723fb625f3c996d Oct 30 12:41:06.984583 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:06.984536 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d"] Oct 30 12:41:07.091267 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:07.091227 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d42ea5c-ec5b-433f-887a-76d2b8a68f46" path="/var/lib/kubelet/pods/9d42ea5c-ec5b-433f-887a-76d2b8a68f46/volumes" Oct 30 12:41:07.412774 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:07.412735 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" event={"ID":"db33d91a-07fb-4c52-832e-0440bc4654af","Type":"ContainerStarted","Data":"c27559fc290f639511e881027f3e747348dcd34f6f375bbf2ebb0009c8e11048"} Oct 30 12:41:07.414657 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:07.414618 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" event={"ID":"64964ff4-80ce-4c89-af7b-19d3b24b07f4","Type":"ContainerStarted","Data":"bbc6c33e039e8a82136c266fd96dbdbbdca18454df568a809ccd55c8ac1ea2f1"} Oct 30 12:41:07.414840 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:07.414686 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" Oct 30 12:41:07.414840 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:07.414707 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" event={"ID":"64964ff4-80ce-4c89-af7b-19d3b24b07f4","Type":"ContainerStarted","Data":"ddc664a80c141a6e81630cc081341063dbcd983124e507484e8f29c0e8acc90f"} Oct 30 12:41:07.416568 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:07.416534 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" event={"ID":"56adb9ab-1a6f-497c-8987-33123b8eb06a","Type":"ContainerStarted","Data":"3cbf3e7c21386a0e05a2f6e6f98d616851a35c3d1d7f856032ed485510ffbcb3"} Oct 30 12:41:07.416701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:07.416594 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" event={"ID":"56adb9ab-1a6f-497c-8987-33123b8eb06a","Type":"ContainerStarted","Data":"4c717a82462a50efaf7f90cbf0f5c57dd9f6caea664128875723fb625f3c996d"} Oct 30 12:41:07.416701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:07.416625 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" Oct 30 12:41:07.458022 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:07.457965 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" podStartSLOduration=1.457952497 podStartE2EDuration="1.457952497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:07.457711718 +0000 UTC m=+2818.491850113" watchObservedRunningTime="2026-10-30 12:41:07.457952497 +0000 UTC m=+2818.492090893" Oct 30 12:41:07.501034 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:07.500980 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" podStartSLOduration=1.500967256 podStartE2EDuration="1.500967256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:07.499701994 +0000 UTC m=+2818.533840392" watchObservedRunningTime="2026-10-30 12:41:07.500967256 +0000 UTC m=+2818.535105721" Oct 30 12:41:07.815215 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:07.815134 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:07.832315 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:07.832275 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx"] Oct 30 12:41:07.832516 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:07.832486 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" podUID="e286532f-d8eb-4c7f-baea-7c71dab94e6e" containerName="istio-proxy" containerID="cri-o://b83423af657a31b3f5749031924bd28e6d43c5006a02fdd6d60c512918a2b549" gracePeriod=30 Oct 30 12:41:08.425867 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:08.425829 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:08.426804 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:08.426786 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:41:10.760301 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:10.760261 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf"] Oct 30 12:41:10.764803 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:10.764777 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf" Oct 30 12:41:10.787547 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:10.787278 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf"] Oct 30 12:41:10.793839 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:10.793800 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-6ad1da\"" Oct 30 12:41:10.800920 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:10.800884 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8czh\" (UniqueName: \"kubernetes.io/projected/52833cb7-6657-4225-9bf4-3532bcd3a15f-kube-api-access-d8czh\") pod \"payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf\" (UID: \"52833cb7-6657-4225-9bf4-3532bcd3a15f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf" Oct 30 12:41:10.801223 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:10.801158 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52833cb7-6657-4225-9bf4-3532bcd3a15f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf\" (UID: \"52833cb7-6657-4225-9bf4-3532bcd3a15f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf" Oct 30 12:41:10.829745 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:10.829243 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2"] Oct 30 12:41:10.835732 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:10.835706 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2" Oct 30 12:41:10.872366 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:10.848354 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2"] Oct 30 12:41:10.902241 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:10.902203 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp944\" (UniqueName: \"kubernetes.io/projected/3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8-kube-api-access-fp944\") pod \"payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2\" (UID: \"3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2" Oct 30 12:41:10.902533 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:10.902511 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52833cb7-6657-4225-9bf4-3532bcd3a15f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf\" (UID: \"52833cb7-6657-4225-9bf4-3532bcd3a15f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf" Oct 30 12:41:10.903200 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:10.903172 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2\" (UID: \"3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2" Oct 30 12:41:10.903430 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:10.903413 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8czh\" (UniqueName: \"kubernetes.io/projected/52833cb7-6657-4225-9bf4-3532bcd3a15f-kube-api-access-d8czh\") pod \"payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf\" (UID: \"52833cb7-6657-4225-9bf4-3532bcd3a15f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf" Oct 30 12:41:10.904979 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:10.904952 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52833cb7-6657-4225-9bf4-3532bcd3a15f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf\" (UID: \"52833cb7-6657-4225-9bf4-3532bcd3a15f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf" Oct 30 12:41:10.955092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:10.941637 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-6ad1da-dockercfg-q5z6z\"" Oct 30 12:41:11.017006 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:11.016921 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8czh\" (UniqueName: \"kubernetes.io/projected/52833cb7-6657-4225-9bf4-3532bcd3a15f-kube-api-access-d8czh\") pod \"payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf\" (UID: \"52833cb7-6657-4225-9bf4-3532bcd3a15f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf" Oct 30 12:41:11.050433 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:11.010056 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp944\" (UniqueName: \"kubernetes.io/projected/3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8-kube-api-access-fp944\") pod \"payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2\" (UID: \"3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2" Oct 30 12:41:11.051179 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:11.051140 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2\" (UID: \"3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2" Oct 30 12:41:11.057103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:11.052343 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2\" (UID: \"3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2" Oct 30 12:41:11.090346 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:11.090310 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf" Oct 30 12:41:11.095800 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:11.095768 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp944\" (UniqueName: \"kubernetes.io/projected/3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8-kube-api-access-fp944\") pod \"payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2\" (UID: \"3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2" Oct 30 12:41:11.162505 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:11.162470 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2" Oct 30 12:41:11.413893 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:41:11.413859 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52833cb7_6657_4225_9bf4_3532bcd3a15f.slice/crio-45397db40ae38195b4aa9a68949c515fd97ffee7696fab54dd389628ce02cfcd WatchSource:0}: Error finding container 45397db40ae38195b4aa9a68949c515fd97ffee7696fab54dd389628ce02cfcd: Status 404 returned error can't find the container with id 45397db40ae38195b4aa9a68949c515fd97ffee7696fab54dd389628ce02cfcd Oct 30 12:41:11.417406 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:11.417382 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf"] Oct 30 12:41:11.450568 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:11.450517 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf" event={"ID":"52833cb7-6657-4225-9bf4-3532bcd3a15f","Type":"ContainerStarted","Data":"45397db40ae38195b4aa9a68949c515fd97ffee7696fab54dd389628ce02cfcd"} Oct 30 12:41:11.490619 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:41:11.490566 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2f93b2_9ab8_4f16_afbb_6d5092ac2ba8.slice/crio-8f8de95dc324a14ff98dd03d10cefd3fb49879a3325e04e87a77181f9e405658 WatchSource:0}: Error finding container 8f8de95dc324a14ff98dd03d10cefd3fb49879a3325e04e87a77181f9e405658: Status 404 returned error can't find the container with id 8f8de95dc324a14ff98dd03d10cefd3fb49879a3325e04e87a77181f9e405658 Oct 30 12:41:11.494033 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:11.494003 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2"] Oct 30 12:41:12.424396 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.424357 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs"] Oct 30 12:41:12.429083 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.429050 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" Oct 30 12:41:12.448523 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.448489 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs"] Oct 30 12:41:12.454627 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.454590 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-2545c7-dockercfg-njtvz\"" Oct 30 12:41:12.454822 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.454741 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-2545c7\"" Oct 30 12:41:12.471304 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.471244 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2" event={"ID":"3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8","Type":"ContainerStarted","Data":"2bfba23f4da0441dc5efe0cdf54fdc93e5242a3c1fbc7ca13fd0e9945a20df8a"} Oct 30 12:41:12.471891 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.471848 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2" event={"ID":"3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8","Type":"ContainerStarted","Data":"8f8de95dc324a14ff98dd03d10cefd3fb49879a3325e04e87a77181f9e405658"} Oct 30 12:41:12.471891 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.471892 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2" Oct 30 12:41:12.474333 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.473409 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf" event={"ID":"52833cb7-6657-4225-9bf4-3532bcd3a15f","Type":"ContainerStarted","Data":"67eca54fa8307cbe89997d3af4172c12b133ddb20ae22c5cfa4cf2285d227ec0"} Oct 30 12:41:12.474333 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.474276 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf" Oct 30 12:41:12.477631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.477597 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bwxt\" (UniqueName: \"kubernetes.io/projected/bc9a08f6-1257-4e45-9a7d-9687d6554377-kube-api-access-2bwxt\") pod \"payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs\" (UID: \"bc9a08f6-1257-4e45-9a7d-9687d6554377\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" Oct 30 12:41:12.477813 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.477664 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9a08f6-1257-4e45-9a7d-9687d6554377-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs\" (UID: \"bc9a08f6-1257-4e45-9a7d-9687d6554377\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" Oct 30 12:41:12.478335 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.478232 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2"] Oct 30 12:41:12.484822 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.484787 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" Oct 30 12:41:12.511444 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.511410 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2"] Oct 30 12:41:12.597985 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.597880 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbt2w\" (UniqueName: \"kubernetes.io/projected/5db888ab-7750-41f1-9a4c-3add0f15cc6d-kube-api-access-qbt2w\") pod \"payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2\" (UID: \"5db888ab-7750-41f1-9a4c-3add0f15cc6d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" Oct 30 12:41:12.598588 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.598561 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bwxt\" (UniqueName: \"kubernetes.io/projected/bc9a08f6-1257-4e45-9a7d-9687d6554377-kube-api-access-2bwxt\") pod \"payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs\" (UID: \"bc9a08f6-1257-4e45-9a7d-9687d6554377\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" Oct 30 12:41:12.599154 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.599133 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9a08f6-1257-4e45-9a7d-9687d6554377-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs\" (UID: \"bc9a08f6-1257-4e45-9a7d-9687d6554377\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" Oct 30 12:41:12.600596 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.600572 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db888ab-7750-41f1-9a4c-3add0f15cc6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2\" (UID: \"5db888ab-7750-41f1-9a4c-3add0f15cc6d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" Oct 30 12:41:12.602823 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.600163 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9a08f6-1257-4e45-9a7d-9687d6554377-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs\" (UID: \"bc9a08f6-1257-4e45-9a7d-9687d6554377\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" Oct 30 12:41:12.679687 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.677064 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2" podStartSLOduration=2.677041385 podStartE2EDuration="2.677041385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:12.531892313 +0000 UTC m=+2823.566030709" watchObservedRunningTime="2026-10-30 12:41:12.677041385 +0000 UTC m=+2823.711179782" Oct 30 12:41:12.704202 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.684649 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf" podStartSLOduration=2.684627908 podStartE2EDuration="2.684627908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:12.604291849 +0000 UTC m=+2823.638430263" watchObservedRunningTime="2026-10-30 12:41:12.684627908 +0000 UTC m=+2823.718766304" Oct 30 12:41:12.706253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.705755 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db888ab-7750-41f1-9a4c-3add0f15cc6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2\" (UID: \"5db888ab-7750-41f1-9a4c-3add0f15cc6d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" Oct 30 12:41:12.706253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.705843 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbt2w\" (UniqueName: \"kubernetes.io/projected/5db888ab-7750-41f1-9a4c-3add0f15cc6d-kube-api-access-qbt2w\") pod \"payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2\" (UID: \"5db888ab-7750-41f1-9a4c-3add0f15cc6d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" Oct 30 12:41:12.707624 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.707569 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db888ab-7750-41f1-9a4c-3add0f15cc6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2\" (UID: \"5db888ab-7750-41f1-9a4c-3add0f15cc6d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" Oct 30 12:41:12.773739 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.749554 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bwxt\" (UniqueName: \"kubernetes.io/projected/bc9a08f6-1257-4e45-9a7d-9687d6554377-kube-api-access-2bwxt\") pod \"payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs\" (UID: \"bc9a08f6-1257-4e45-9a7d-9687d6554377\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" Oct 30 12:41:12.773739 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.773423 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" Oct 30 12:41:12.810212 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.800558 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbt2w\" (UniqueName: \"kubernetes.io/projected/5db888ab-7750-41f1-9a4c-3add0f15cc6d-kube-api-access-qbt2w\") pod \"payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2\" (UID: \"5db888ab-7750-41f1-9a4c-3add0f15cc6d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" Oct 30 12:41:12.835115 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:12.824742 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" Oct 30 12:41:13.143339 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.143306 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:41:13.179772 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:41:13.179736 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db888ab_7750_41f1_9a4c_3add0f15cc6d.slice/crio-4b91a306c006f50aa68f1efbd8afd3b1847ed923f49507beeb2c6d0ff9540dbb WatchSource:0}: Error finding container 4b91a306c006f50aa68f1efbd8afd3b1847ed923f49507beeb2c6d0ff9540dbb: Status 404 returned error can't find the container with id 4b91a306c006f50aa68f1efbd8afd3b1847ed923f49507beeb2c6d0ff9540dbb Oct 30 12:41:13.183514 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.183447 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2"] Oct 30 12:41:13.193200 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.193170 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs"] Oct 30 12:41:13.225010 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.224954 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-workload-socket\") pod \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " Oct 30 12:41:13.225244 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.225027 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e286532f-d8eb-4c7f-baea-7c71dab94e6e-kube-api-access-v4zzn\" (UniqueName: \"kubernetes.io/projected/e286532f-d8eb-4c7f-baea-7c71dab94e6e-kube-api-access-v4zzn\") pod \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " Oct 30 12:41:13.225244 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.225087 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-data\") pod \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " Oct 30 12:41:13.225424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.225397 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-workload-certs\") pod \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " Oct 30 12:41:13.225556 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.225440 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-credential-socket\") pod \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " Oct 30 12:41:13.225556 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.225485 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-envoy\") pod \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " Oct 30 12:41:13.225556 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.225511 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-token\" (UniqueName: \"kubernetes.io/projected/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-token\") pod \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " Oct 30 12:41:13.225715 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.225564 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istiod-ca-cert\") pod \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " Oct 30 12:41:13.225715 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.225597 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-podinfo\") pod \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\" (UID: \"e286532f-d8eb-4c7f-baea-7c71dab94e6e\") " Oct 30 12:41:13.225854 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.225830 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-data" pod "e286532f-d8eb-4c7f-baea-7c71dab94e6e" (UID: "e286532f-d8eb-4c7f-baea-7c71dab94e6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:13.227109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.226259 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:13.227109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.226406 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-workload-certs" pod "e286532f-d8eb-4c7f-baea-7c71dab94e6e" (UID: "e286532f-d8eb-4c7f-baea-7c71dab94e6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:13.227109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.226420 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-workload-socket" pod "e286532f-d8eb-4c7f-baea-7c71dab94e6e" (UID: "e286532f-d8eb-4c7f-baea-7c71dab94e6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:13.227109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.226635 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-credential-socket" pod "e286532f-d8eb-4c7f-baea-7c71dab94e6e" (UID: "e286532f-d8eb-4c7f-baea-7c71dab94e6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:13.227109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.226975 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istiod-ca-cert" pod "e286532f-d8eb-4c7f-baea-7c71dab94e6e" (UID: "e286532f-d8eb-4c7f-baea-7c71dab94e6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:41:13.229267 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.229216 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-envoy" pod "e286532f-d8eb-4c7f-baea-7c71dab94e6e" (UID: "e286532f-d8eb-4c7f-baea-7c71dab94e6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:13.229531 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.229501 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-podinfo" pod "e286532f-d8eb-4c7f-baea-7c71dab94e6e" (UID: "e286532f-d8eb-4c7f-baea-7c71dab94e6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:41:13.229531 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.229516 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e286532f-d8eb-4c7f-baea-7c71dab94e6e-kube-api-access-v4zzn" pod "e286532f-d8eb-4c7f-baea-7c71dab94e6e" (UID: "e286532f-d8eb-4c7f-baea-7c71dab94e6e"). InnerVolumeSpecName "kube-api-access-v4zzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:13.230490 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.230456 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-token" pod "e286532f-d8eb-4c7f-baea-7c71dab94e6e" (UID: "e286532f-d8eb-4c7f-baea-7c71dab94e6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:13.327521 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.327488 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:13.327521 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.327519 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:13.327737 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.327531 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:13.327737 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.327545 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4zzn\" (UniqueName: \"kubernetes.io/projected/e286532f-d8eb-4c7f-baea-7c71dab94e6e-kube-api-access-v4zzn\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:13.327737 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.327556 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:13.327737 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.327564 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:13.327737 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.327575 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:13.327737 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.327584 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e286532f-d8eb-4c7f-baea-7c71dab94e6e-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:13.483373 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.483279 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" event={"ID":"bc9a08f6-1257-4e45-9a7d-9687d6554377","Type":"ContainerStarted","Data":"26c1b443cb3f7f1effe37bd0a04f9878fb15d08bfa54d811287c7dd53f089a87"} Oct 30 12:41:13.483373 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.483338 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" event={"ID":"bc9a08f6-1257-4e45-9a7d-9687d6554377","Type":"ContainerStarted","Data":"2b61bef3798f96a691cdbdb5fe122836f67bf6246e0a4a79d8b0a880ec28945d"} Oct 30 12:41:13.484410 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.483481 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" Oct 30 12:41:13.484948 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.484925 4468 generic.go:356] "Generic (PLEG): container finished" podID="e286532f-d8eb-4c7f-baea-7c71dab94e6e" containerID="b83423af657a31b3f5749031924bd28e6d43c5006a02fdd6d60c512918a2b549" exitCode=0 Oct 30 12:41:13.485022 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.484986 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" Oct 30 12:41:13.485061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.485013 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" event={"ID":"e286532f-d8eb-4c7f-baea-7c71dab94e6e","Type":"ContainerDied","Data":"b83423af657a31b3f5749031924bd28e6d43c5006a02fdd6d60c512918a2b549"} Oct 30 12:41:13.485170 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.485107 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx" event={"ID":"e286532f-d8eb-4c7f-baea-7c71dab94e6e","Type":"ContainerDied","Data":"43b3279e8d46cb7449157ed63b5e136d6fa7e52a52fe89915013c6ff6cc6aa16"} Oct 30 12:41:13.485170 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.485148 4468 scope.go:122] "RemoveContainer" containerID="b83423af657a31b3f5749031924bd28e6d43c5006a02fdd6d60c512918a2b549" Oct 30 12:41:13.486969 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.486930 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" event={"ID":"5db888ab-7750-41f1-9a4c-3add0f15cc6d","Type":"ContainerStarted","Data":"0dabc8ba218f01b32a71860b65add6d6de98fcb2ed635627e028295738d7e7b5"} Oct 30 12:41:13.487138 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.487046 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" event={"ID":"5db888ab-7750-41f1-9a4c-3add0f15cc6d","Type":"ContainerStarted","Data":"4b91a306c006f50aa68f1efbd8afd3b1847ed923f49507beeb2c6d0ff9540dbb"} Oct 30 12:41:13.487859 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.487403 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" Oct 30 12:41:13.497241 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.497216 4468 scope.go:122] "RemoveContainer" containerID="b83423af657a31b3f5749031924bd28e6d43c5006a02fdd6d60c512918a2b549" Oct 30 12:41:13.497618 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:41:13.497594 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b83423af657a31b3f5749031924bd28e6d43c5006a02fdd6d60c512918a2b549\": container with ID starting with b83423af657a31b3f5749031924bd28e6d43c5006a02fdd6d60c512918a2b549 not found: ID does not exist" containerID="b83423af657a31b3f5749031924bd28e6d43c5006a02fdd6d60c512918a2b549" Oct 30 12:41:13.497689 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.497626 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b83423af657a31b3f5749031924bd28e6d43c5006a02fdd6d60c512918a2b549"} err="failed to get container status \"b83423af657a31b3f5749031924bd28e6d43c5006a02fdd6d60c512918a2b549\": rpc error: code = NotFound desc = could not find container \"b83423af657a31b3f5749031924bd28e6d43c5006a02fdd6d60c512918a2b549\": container with ID starting with b83423af657a31b3f5749031924bd28e6d43c5006a02fdd6d60c512918a2b549 not found: ID does not exist" Oct 30 12:41:13.532505 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.532409 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" podStartSLOduration=1.532371716 podStartE2EDuration="1.532371716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:13.527142139 +0000 UTC m=+2824.561280536" watchObservedRunningTime="2026-10-30 12:41:13.532371716 +0000 UTC m=+2824.566510116" Oct 30 12:41:13.570767 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.570708 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx"] Oct 30 12:41:13.575508 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.575466 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-p2gnx"] Oct 30 12:41:13.614113 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:13.614021 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" podStartSLOduration=1.6140028480000002 podStartE2EDuration="1.614002848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:13.610297453 +0000 UTC m=+2824.644435848" watchObservedRunningTime="2026-10-30 12:41:13.614002848 +0000 UTC m=+2824.648141243" Oct 30 12:41:15.092705 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:15.092668 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e286532f-d8eb-4c7f-baea-7c71dab94e6e" path="/var/lib/kubelet/pods/e286532f-d8eb-4c7f-baea-7c71dab94e6e/volumes" Oct 30 12:41:18.428562 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:18.428531 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" Oct 30 12:41:18.429082 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:18.428611 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" Oct 30 12:41:22.839855 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:22.839768 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4"] Oct 30 12:41:22.840372 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:22.840030 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" podUID="bdc9d563-a787-4dac-b192-9507eaad1b14" containerName="istio-proxy" containerID="cri-o://ef8f4ee77bc4a535e1043d82cab03666d5e5b5c1d0d1f55e7db7ab627aca7bea" gracePeriod=30 Oct 30 12:41:23.490989 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:23.490953 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2" Oct 30 12:41:23.602058 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:41:23.602011 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:41:24.509092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:24.507971 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" Oct 30 12:41:24.513170 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:24.509908 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf" Oct 30 12:41:24.513170 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:24.511627 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" Oct 30 12:41:28.179039 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.179009 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:41:28.272548 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.272472 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-workload-socket\") pod \"bdc9d563-a787-4dac-b192-9507eaad1b14\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " Oct 30 12:41:28.272781 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.272625 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-token\" (UniqueName: \"kubernetes.io/projected/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-token\") pod \"bdc9d563-a787-4dac-b192-9507eaad1b14\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " Oct 30 12:41:28.272781 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.272658 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdc9d563-a787-4dac-b192-9507eaad1b14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdc9d563-a787-4dac-b192-9507eaad1b14-istiod-ca-cert\") pod \"bdc9d563-a787-4dac-b192-9507eaad1b14\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " Oct 30 12:41:28.272781 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.272741 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-credential-socket\") pod \"bdc9d563-a787-4dac-b192-9507eaad1b14\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " Oct 30 12:41:28.273053 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.272809 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc9d563-a787-4dac-b192-9507eaad1b14-kube-api-access-xzw2w\" (UniqueName: \"kubernetes.io/projected/bdc9d563-a787-4dac-b192-9507eaad1b14-kube-api-access-xzw2w\") pod \"bdc9d563-a787-4dac-b192-9507eaad1b14\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " Oct 30 12:41:28.273053 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.272893 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-data\") pod \"bdc9d563-a787-4dac-b192-9507eaad1b14\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " Oct 30 12:41:28.273257 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.273225 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-credential-socket" pod "bdc9d563-a787-4dac-b192-9507eaad1b14" (UID: "bdc9d563-a787-4dac-b192-9507eaad1b14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:28.273389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.273352 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-envoy\") pod \"bdc9d563-a787-4dac-b192-9507eaad1b14\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " Oct 30 12:41:28.273465 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.273439 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-workload-certs\") pod \"bdc9d563-a787-4dac-b192-9507eaad1b14\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " Oct 30 12:41:28.273527 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.273509 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-workload-socket" pod "bdc9d563-a787-4dac-b192-9507eaad1b14" (UID: "bdc9d563-a787-4dac-b192-9507eaad1b14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:28.273527 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.273520 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-podinfo\") pod \"bdc9d563-a787-4dac-b192-9507eaad1b14\" (UID: \"bdc9d563-a787-4dac-b192-9507eaad1b14\") " Oct 30 12:41:28.273654 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.273635 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-data" pod "bdc9d563-a787-4dac-b192-9507eaad1b14" (UID: "bdc9d563-a787-4dac-b192-9507eaad1b14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:28.273749 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.273729 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-workload-certs" pod "bdc9d563-a787-4dac-b192-9507eaad1b14" (UID: "bdc9d563-a787-4dac-b192-9507eaad1b14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:28.273974 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.273831 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc9d563-a787-4dac-b192-9507eaad1b14-istiod-ca-cert" pod "bdc9d563-a787-4dac-b192-9507eaad1b14" (UID: "bdc9d563-a787-4dac-b192-9507eaad1b14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:41:28.274401 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.274377 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:28.274508 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.274404 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:28.274508 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.274420 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdc9d563-a787-4dac-b192-9507eaad1b14-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:28.274508 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.274434 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:28.274508 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.274447 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:28.275324 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.275294 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-token" pod "bdc9d563-a787-4dac-b192-9507eaad1b14" (UID: "bdc9d563-a787-4dac-b192-9507eaad1b14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:28.275827 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.275801 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-envoy" pod "bdc9d563-a787-4dac-b192-9507eaad1b14" (UID: "bdc9d563-a787-4dac-b192-9507eaad1b14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:28.275895 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.275821 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc9d563-a787-4dac-b192-9507eaad1b14-kube-api-access-xzw2w" pod "bdc9d563-a787-4dac-b192-9507eaad1b14" (UID: "bdc9d563-a787-4dac-b192-9507eaad1b14"). InnerVolumeSpecName "kube-api-access-xzw2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:28.275933 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.275893 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-podinfo" pod "bdc9d563-a787-4dac-b192-9507eaad1b14" (UID: "bdc9d563-a787-4dac-b192-9507eaad1b14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:41:28.340163 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.340121 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts"] Oct 30 12:41:28.340874 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.340851 4468 cpu_manager.go:414] "containerMap: removing container" podUID="bdc9d563-a787-4dac-b192-9507eaad1b14" containerName="istio-proxy" Oct 30 12:41:28.340874 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.340875 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc9d563-a787-4dac-b192-9507eaad1b14" containerName="istio-proxy" Oct 30 12:41:28.341086 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.340945 4468 cpu_manager.go:414] "containerMap: removing container" podUID="e286532f-d8eb-4c7f-baea-7c71dab94e6e" containerName="istio-proxy" Oct 30 12:41:28.341086 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.340955 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e286532f-d8eb-4c7f-baea-7c71dab94e6e" containerName="istio-proxy" Oct 30 12:41:28.341204 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.341163 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc9d563-a787-4dac-b192-9507eaad1b14" containerName="istio-proxy" Oct 30 12:41:28.341204 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.341191 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="e286532f-d8eb-4c7f-baea-7c71dab94e6e" containerName="istio-proxy" Oct 30 12:41:28.354959 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.345809 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.368529 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.368491 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-f7464f-openshift-default-dockercfg-ccx4c\"" Oct 30 12:41:28.370543 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.370511 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts"] Oct 30 12:41:28.376138 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.376038 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzw2w\" (UniqueName: \"kubernetes.io/projected/bdc9d563-a787-4dac-b192-9507eaad1b14-kube-api-access-xzw2w\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:28.376138 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.376088 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:28.376138 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.376103 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:28.376138 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.376114 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdc9d563-a787-4dac-b192-9507eaad1b14-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:41:28.480089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.477249 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-workload-certs\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.480089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.477318 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bpx6\" (UniqueName: \"kubernetes.io/projected/0b15b44d-dfe6-4001-966b-f72c56f449c1-kube-api-access-5bpx6\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.480089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.477402 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-envoy\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.480089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.477555 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-podinfo\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.480089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.477604 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-workload-socket\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.480089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.477657 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b15b44d-dfe6-4001-966b-f72c56f449c1-istiod-ca-cert\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.480089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.477718 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-credential-socket\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.480089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.477739 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-data\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.480089 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.477885 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-token\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.582092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.580698 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-token\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.582092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.580802 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-workload-certs\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.582092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.580851 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bpx6\" (UniqueName: \"kubernetes.io/projected/0b15b44d-dfe6-4001-966b-f72c56f449c1-kube-api-access-5bpx6\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.582092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.580951 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-envoy\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.582092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.581052 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-podinfo\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.582092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.581118 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-workload-socket\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.582092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.581164 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b15b44d-dfe6-4001-966b-f72c56f449c1-istiod-ca-cert\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.582092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.581318 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-credential-socket\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.582092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.581346 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-data\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.582092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.581886 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-data\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.583181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.582603 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-workload-certs\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.583503 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.583477 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-workload-socket\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.584805 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.584445 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-credential-socket\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.587768 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.585412 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b15b44d-dfe6-4001-966b-f72c56f449c1-istiod-ca-cert\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.592353 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.592313 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-envoy\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.593117 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.592708 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-podinfo\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.606844 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.606809 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-token\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.606844 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.606828 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bpx6\" (UniqueName: \"kubernetes.io/projected/0b15b44d-dfe6-4001-966b-f72c56f449c1-kube-api-access-5bpx6\") pod \"e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.608036 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.607992 4468 generic.go:356] "Generic (PLEG): container finished" podID="bdc9d563-a787-4dac-b192-9507eaad1b14" containerID="ef8f4ee77bc4a535e1043d82cab03666d5e5b5c1d0d1f55e7db7ab627aca7bea" exitCode=0 Oct 30 12:41:28.608252 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.608230 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" event={"ID":"bdc9d563-a787-4dac-b192-9507eaad1b14","Type":"ContainerDied","Data":"ef8f4ee77bc4a535e1043d82cab03666d5e5b5c1d0d1f55e7db7ab627aca7bea"} Oct 30 12:41:28.608348 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.608290 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" event={"ID":"bdc9d563-a787-4dac-b192-9507eaad1b14","Type":"ContainerDied","Data":"4585c0efe745ed402996841556a4dcb00c2bcab44a589e9ab41edc050c7850c9"} Oct 30 12:41:28.608425 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.608323 4468 scope.go:122] "RemoveContainer" containerID="ef8f4ee77bc4a535e1043d82cab03666d5e5b5c1d0d1f55e7db7ab627aca7bea" Oct 30 12:41:28.608495 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.608483 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4" Oct 30 12:41:28.644189 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.643457 4468 scope.go:122] "RemoveContainer" containerID="ef8f4ee77bc4a535e1043d82cab03666d5e5b5c1d0d1f55e7db7ab627aca7bea" Oct 30 12:41:28.644189 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:41:28.643957 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8f4ee77bc4a535e1043d82cab03666d5e5b5c1d0d1f55e7db7ab627aca7bea\": container with ID starting with ef8f4ee77bc4a535e1043d82cab03666d5e5b5c1d0d1f55e7db7ab627aca7bea not found: ID does not exist" containerID="ef8f4ee77bc4a535e1043d82cab03666d5e5b5c1d0d1f55e7db7ab627aca7bea" Oct 30 12:41:28.644189 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.643998 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8f4ee77bc4a535e1043d82cab03666d5e5b5c1d0d1f55e7db7ab627aca7bea"} err="failed to get container status \"ef8f4ee77bc4a535e1043d82cab03666d5e5b5c1d0d1f55e7db7ab627aca7bea\": rpc error: code = NotFound desc = could not find container \"ef8f4ee77bc4a535e1043d82cab03666d5e5b5c1d0d1f55e7db7ab627aca7bea\": container with ID starting with ef8f4ee77bc4a535e1043d82cab03666d5e5b5c1d0d1f55e7db7ab627aca7bea not found: ID does not exist" Oct 30 12:41:28.674504 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.674459 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4"] Oct 30 12:41:28.683706 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.683653 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:28.694331 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:28.694295 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-2vgx4"] Oct 30 12:41:29.011027 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:41:29.010989 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b15b44d_dfe6_4001_966b_f72c56f449c1.slice/crio-4870ec8c883e210108ddc1af4e14c2c5d278b8bac26833d3b07f4c1de41df14c WatchSource:0}: Error finding container 4870ec8c883e210108ddc1af4e14c2c5d278b8bac26833d3b07f4c1de41df14c: Status 404 returned error can't find the container with id 4870ec8c883e210108ddc1af4e14c2c5d278b8bac26833d3b07f4c1de41df14c Oct 30 12:41:29.013848 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:29.013820 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts"] Oct 30 12:41:29.015120 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:29.015035 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:29.015217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:29.015180 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:29.015285 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:29.015230 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:41:29.095349 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:29.095318 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdc9d563-a787-4dac-b192-9507eaad1b14" path="/var/lib/kubelet/pods/bdc9d563-a787-4dac-b192-9507eaad1b14/volumes" Oct 30 12:41:29.617466 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:29.617426 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" event={"ID":"0b15b44d-dfe6-4001-966b-f72c56f449c1","Type":"ContainerStarted","Data":"c26876500b1851d4d59a53365d3ee95f0d5a608feb2148d066743c2686108c5c"} Oct 30 12:41:29.617919 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:29.617496 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" event={"ID":"0b15b44d-dfe6-4001-966b-f72c56f449c1","Type":"ContainerStarted","Data":"4870ec8c883e210108ddc1af4e14c2c5d278b8bac26833d3b07f4c1de41df14c"} Oct 30 12:41:29.665733 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:29.665679 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" podStartSLOduration=1.6656642609999999 podStartE2EDuration="1.665664261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:29.663946079 +0000 UTC m=+2840.698084484" watchObservedRunningTime="2026-10-30 12:41:29.665664261 +0000 UTC m=+2840.699802657" Oct 30 12:41:29.683867 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:29.683837 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:30.690514 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:30.690476 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:31.631339 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:31.631307 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:31.632545 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:31.632526 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:41:34.333888 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.333845 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r"] Oct 30 12:41:34.338846 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.338821 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" Oct 30 12:41:34.359415 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.359380 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r"] Oct 30 12:41:34.366681 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.366647 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-f7464f\"" Oct 30 12:41:34.388815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.388771 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9"] Oct 30 12:41:34.394733 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.394702 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" Oct 30 12:41:34.411594 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.411556 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-f7464f-dockercfg-24vc2\"" Oct 30 12:41:34.413992 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.413957 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9"] Oct 30 12:41:34.479888 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.478569 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkbng\" (UniqueName: \"kubernetes.io/projected/5ae0dc66-24bc-4aad-a275-bc7a8a2c3206-kube-api-access-tkbng\") pod \"payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9\" (UID: \"5ae0dc66-24bc-4aad-a275-bc7a8a2c3206\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" Oct 30 12:41:34.479888 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.478627 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae0dc66-24bc-4aad-a275-bc7a8a2c3206-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9\" (UID: \"5ae0dc66-24bc-4aad-a275-bc7a8a2c3206\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" Oct 30 12:41:34.479888 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.478782 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpfbt\" (UniqueName: \"kubernetes.io/projected/7f2bdcd0-98ce-4c5b-bb1b-81db35f39186-kube-api-access-tpfbt\") pod \"payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r\" (UID: \"7f2bdcd0-98ce-4c5b-bb1b-81db35f39186\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" Oct 30 12:41:34.479888 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.478856 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f2bdcd0-98ce-4c5b-bb1b-81db35f39186-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r\" (UID: \"7f2bdcd0-98ce-4c5b-bb1b-81db35f39186\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" Oct 30 12:41:34.593142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.582926 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpfbt\" (UniqueName: \"kubernetes.io/projected/7f2bdcd0-98ce-4c5b-bb1b-81db35f39186-kube-api-access-tpfbt\") pod \"payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r\" (UID: \"7f2bdcd0-98ce-4c5b-bb1b-81db35f39186\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" Oct 30 12:41:34.593142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.583041 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f2bdcd0-98ce-4c5b-bb1b-81db35f39186-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r\" (UID: \"7f2bdcd0-98ce-4c5b-bb1b-81db35f39186\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" Oct 30 12:41:34.593142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.583444 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkbng\" (UniqueName: \"kubernetes.io/projected/5ae0dc66-24bc-4aad-a275-bc7a8a2c3206-kube-api-access-tkbng\") pod \"payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9\" (UID: \"5ae0dc66-24bc-4aad-a275-bc7a8a2c3206\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" Oct 30 12:41:34.593142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.583488 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae0dc66-24bc-4aad-a275-bc7a8a2c3206-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9\" (UID: \"5ae0dc66-24bc-4aad-a275-bc7a8a2c3206\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" Oct 30 12:41:34.593142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.584622 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae0dc66-24bc-4aad-a275-bc7a8a2c3206-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9\" (UID: \"5ae0dc66-24bc-4aad-a275-bc7a8a2c3206\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" Oct 30 12:41:34.593142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.585857 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f2bdcd0-98ce-4c5b-bb1b-81db35f39186-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r\" (UID: \"7f2bdcd0-98ce-4c5b-bb1b-81db35f39186\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" Oct 30 12:41:34.634094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.631163 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpfbt\" (UniqueName: \"kubernetes.io/projected/7f2bdcd0-98ce-4c5b-bb1b-81db35f39186-kube-api-access-tpfbt\") pod \"payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r\" (UID: \"7f2bdcd0-98ce-4c5b-bb1b-81db35f39186\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" Oct 30 12:41:34.641091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.638581 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkbng\" (UniqueName: \"kubernetes.io/projected/5ae0dc66-24bc-4aad-a275-bc7a8a2c3206-kube-api-access-tkbng\") pod \"payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9\" (UID: \"5ae0dc66-24bc-4aad-a275-bc7a8a2c3206\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" Oct 30 12:41:34.698185 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.696310 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" Oct 30 12:41:34.728149 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:34.715129 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" Oct 30 12:41:35.092747 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:35.092718 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9"] Oct 30 12:41:35.100464 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:41:35.097154 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae0dc66_24bc_4aad_a275_bc7a8a2c3206.slice/crio-a0653dd9e5b8c39838529534778d9b2ce8ec1d7568a3213b555359979c5e2589 WatchSource:0}: Error finding container a0653dd9e5b8c39838529534778d9b2ce8ec1d7568a3213b555359979c5e2589: Status 404 returned error can't find the container with id a0653dd9e5b8c39838529534778d9b2ce8ec1d7568a3213b555359979c5e2589 Oct 30 12:41:35.106767 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:35.106739 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r"] Oct 30 12:41:35.748673 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:35.748630 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" event={"ID":"5ae0dc66-24bc-4aad-a275-bc7a8a2c3206","Type":"ContainerStarted","Data":"b430a7f1d24b7e8e908d15f04298d8f415b2cda5bdc4a08e350a9fc9189ea17a"} Oct 30 12:41:35.749202 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:35.748695 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" Oct 30 12:41:35.749202 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:35.748721 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" event={"ID":"5ae0dc66-24bc-4aad-a275-bc7a8a2c3206","Type":"ContainerStarted","Data":"a0653dd9e5b8c39838529534778d9b2ce8ec1d7568a3213b555359979c5e2589"} Oct 30 12:41:35.750463 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:35.750429 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" event={"ID":"7f2bdcd0-98ce-4c5b-bb1b-81db35f39186","Type":"ContainerStarted","Data":"e9e6b3e52ecc3dde7976c2aae31130b82f0188c25a2edff33b924efe68ae99e1"} Oct 30 12:41:35.750587 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:35.750477 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" event={"ID":"7f2bdcd0-98ce-4c5b-bb1b-81db35f39186","Type":"ContainerStarted","Data":"b64347a4e24f4eee4ac9640e6732b3f1f4545bde1439502879eb9a3a70dfd4ae"} Oct 30 12:41:35.750587 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:35.750498 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" Oct 30 12:41:35.793767 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:35.793704 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" podStartSLOduration=1.79369042 podStartE2EDuration="1.79369042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:35.791471376 +0000 UTC m=+2846.825609773" watchObservedRunningTime="2026-10-30 12:41:35.79369042 +0000 UTC m=+2846.827828830" Oct 30 12:41:35.836459 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:35.836403 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" podStartSLOduration=1.836387138 podStartE2EDuration="1.836387138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:35.833528553 +0000 UTC m=+2846.867666949" watchObservedRunningTime="2026-10-30 12:41:35.836387138 +0000 UTC m=+2846.870525534" Oct 30 12:41:46.762820 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:46.762788 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" Oct 30 12:41:46.763365 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:46.762873 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" Oct 30 12:41:47.188721 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:47.188683 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq"] Oct 30 12:41:47.198808 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:47.198723 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq" Oct 30 12:41:47.202533 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:47.202501 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq"] Oct 30 12:41:47.219464 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:47.217824 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-ac674b-dockercfg-tm4xx\"" Oct 30 12:41:47.219464 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:47.218577 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-ac674b\"" Oct 30 12:41:47.375389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:47.375355 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a-plugins-config-volume\") pod \"payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq\" (UID: \"d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a\") " pod="openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq" Oct 30 12:41:47.375561 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:47.375422 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7tvp\" (UniqueName: \"kubernetes.io/projected/d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a-kube-api-access-m7tvp\") pod \"payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq\" (UID: \"d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a\") " pod="openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq" Oct 30 12:41:47.477227 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:47.477134 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a-plugins-config-volume\") pod \"payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq\" (UID: \"d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a\") " pod="openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq" Oct 30 12:41:47.477227 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:47.477206 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7tvp\" (UniqueName: \"kubernetes.io/projected/d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a-kube-api-access-m7tvp\") pod \"payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq\" (UID: \"d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a\") " pod="openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq" Oct 30 12:41:47.478210 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:47.478176 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a-plugins-config-volume\") pod \"payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq\" (UID: \"d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a\") " pod="openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq" Oct 30 12:41:47.502237 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:47.502203 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7tvp\" (UniqueName: \"kubernetes.io/projected/d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a-kube-api-access-m7tvp\") pod \"payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq\" (UID: \"d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a\") " pod="openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq" Oct 30 12:41:47.528836 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:47.528804 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq" Oct 30 12:41:47.839096 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:41:47.839032 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd39dd7e1_64b2_4ec3_a7ea_c94ee9299f3a.slice/crio-11316a9787769b9cf24e0be75bd75bf7613e917d942d552efae70f804b23010b WatchSource:0}: Error finding container 11316a9787769b9cf24e0be75bd75bf7613e917d942d552efae70f804b23010b: Status 404 returned error can't find the container with id 11316a9787769b9cf24e0be75bd75bf7613e917d942d552efae70f804b23010b Oct 30 12:41:47.841373 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:47.841349 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq"] Oct 30 12:41:47.911936 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:47.911899 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq" event={"ID":"d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a","Type":"ContainerStarted","Data":"11316a9787769b9cf24e0be75bd75bf7613e917d942d552efae70f804b23010b"} Oct 30 12:41:48.920343 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:48.920297 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq" event={"ID":"d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a","Type":"ContainerStarted","Data":"22a5cbd77164e9d4e5c9a698770830fa1fb61f047806381c25b5c0436c91756f"} Oct 30 12:41:48.920849 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:48.920379 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq" Oct 30 12:41:48.964161 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:48.964104 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq" podStartSLOduration=1.964087504 podStartE2EDuration="1.964087504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:48.963223135 +0000 UTC m=+2859.997361532" watchObservedRunningTime="2026-10-30 12:41:48.964087504 +0000 UTC m=+2859.998225897" Oct 30 12:41:53.600687 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:41:53.600644 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:41:59.931487 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:41:59.931458 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq" Oct 30 12:42:04.411963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:04.411925 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl"] Oct 30 12:42:04.417793 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:04.417773 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" Oct 30 12:42:04.437675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:04.437644 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl"] Oct 30 12:42:04.438415 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:04.438386 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-f7464f-serving-cert\"" Oct 30 12:42:04.549985 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:04.549934 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-metrics-certs\") pod \"maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl\" (UID: \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" Oct 30 12:42:04.550239 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:04.550115 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljpn7\" (UniqueName: \"kubernetes.io/projected/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-kube-api-access-ljpn7\") pod \"maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl\" (UID: \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" Oct 30 12:42:04.550335 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:04.550315 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl\" (UID: \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" Oct 30 12:42:04.550485 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:04.550468 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl\" (UID: \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" Oct 30 12:42:04.653703 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:04.653664 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljpn7\" (UniqueName: \"kubernetes.io/projected/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-kube-api-access-ljpn7\") pod \"maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl\" (UID: \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" Oct 30 12:42:04.654230 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:04.654170 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl\" (UID: \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" Oct 30 12:42:04.654812 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:04.654775 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl\" (UID: \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" Oct 30 12:42:04.654928 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:04.654894 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-metrics-certs\") pod \"maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl\" (UID: \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" Oct 30 12:42:04.656199 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:04.656157 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl\" (UID: \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" Oct 30 12:42:04.708137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:04.697731 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl\" (UID: \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" Oct 30 12:42:04.708137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:04.707063 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljpn7\" (UniqueName: \"kubernetes.io/projected/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-kube-api-access-ljpn7\") pod \"maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl\" (UID: \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" Oct 30 12:42:04.716347 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:04.716303 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-metrics-certs\") pod \"maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl\" (UID: \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" Oct 30 12:42:04.737339 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:04.737295 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" Oct 30 12:42:05.047403 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:42:05.047365 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1270203a_f11c_4b92_9cb6_b76ab1bbc2a4.slice/crio-5ac6bd22277d62da4e67c6ae9bfb1d6b840b9cbdf6bfe7e740f386cd4cc4cd1e WatchSource:0}: Error finding container 5ac6bd22277d62da4e67c6ae9bfb1d6b840b9cbdf6bfe7e740f386cd4cc4cd1e: Status 404 returned error can't find the container with id 5ac6bd22277d62da4e67c6ae9bfb1d6b840b9cbdf6bfe7e740f386cd4cc4cd1e Oct 30 12:42:05.049997 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:05.049974 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl"] Oct 30 12:42:06.091203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:06.085323 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" event={"ID":"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4","Type":"ContainerStarted","Data":"30026e6efbe060567e62c6138a412e1c204b95ef760d9d376e33914efba76a14"} Oct 30 12:42:06.091203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:06.085394 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" event={"ID":"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4","Type":"ContainerStarted","Data":"5ac6bd22277d62da4e67c6ae9bfb1d6b840b9cbdf6bfe7e740f386cd4cc4cd1e"} Oct 30 12:42:06.091203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:06.086838 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" Oct 30 12:42:06.152832 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:06.152176 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" podStartSLOduration=1.753027388 podStartE2EDuration="2.152151845s" totalImagesPullingTime="399.124457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:06.136529007 +0000 UTC m=+2877.170667406" watchObservedRunningTime="2026-10-30 12:42:06.152151845 +0000 UTC m=+2877.186290232" Oct 30 12:42:13.120136 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:13.120103 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" Oct 30 12:42:18.223782 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:18.223740 4468 generic.go:356] "Generic (PLEG): container finished" podID="db33d91a-07fb-4c52-832e-0440bc4654af" containerID="c27559fc290f639511e881027f3e747348dcd34f6f375bbf2ebb0009c8e11048" exitCode=0 Oct 30 12:42:18.224311 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:18.223792 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" event={"ID":"db33d91a-07fb-4c52-832e-0440bc4654af","Type":"ContainerDied","Data":"c27559fc290f639511e881027f3e747348dcd34f6f375bbf2ebb0009c8e11048"} Oct 30 12:42:18.224420 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:18.224395 4468 scope.go:122] "RemoveContainer" containerID="c27559fc290f639511e881027f3e747348dcd34f6f375bbf2ebb0009c8e11048" Oct 30 12:42:18.226268 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:18.226239 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:18.226360 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:18.226319 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:18.226404 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:18.226360 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:19.195956 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:19.192649 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8"] Oct 30 12:42:19.199858 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:19.199827 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" Oct 30 12:42:19.228710 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:19.228667 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8"] Oct 30 12:42:19.229601 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:19.229579 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0f7d14\"" Oct 30 12:42:19.258457 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:19.258185 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" event={"ID":"db33d91a-07fb-4c52-832e-0440bc4654af","Type":"ContainerStarted","Data":"c3fd1d62eecac0293c9ff5d09a3fb33834c62c4d6a05f3ca5609c2abf1b93114"} Oct 30 12:42:19.270148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:19.269846 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0f7d14-dockercfg-k6ftm\"" Oct 30 12:42:19.341302 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:19.341244 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kr22\" (UniqueName: \"kubernetes.io/projected/69e26174-f25d-4904-b2d8-b69a962ca344-kube-api-access-9kr22\") pod \"payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8\" (UID: \"69e26174-f25d-4904-b2d8-b69a962ca344\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" Oct 30 12:42:19.341552 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:19.341535 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e26174-f25d-4904-b2d8-b69a962ca344-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8\" (UID: \"69e26174-f25d-4904-b2d8-b69a962ca344\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" Oct 30 12:42:19.446300 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:19.446208 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e26174-f25d-4904-b2d8-b69a962ca344-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8\" (UID: \"69e26174-f25d-4904-b2d8-b69a962ca344\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" Oct 30 12:42:19.454106 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:19.451382 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kr22\" (UniqueName: \"kubernetes.io/projected/69e26174-f25d-4904-b2d8-b69a962ca344-kube-api-access-9kr22\") pod \"payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8\" (UID: \"69e26174-f25d-4904-b2d8-b69a962ca344\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" Oct 30 12:42:19.454106 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:19.452422 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e26174-f25d-4904-b2d8-b69a962ca344-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8\" (UID: \"69e26174-f25d-4904-b2d8-b69a962ca344\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" Oct 30 12:42:19.517529 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:19.517494 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kr22\" (UniqueName: \"kubernetes.io/projected/69e26174-f25d-4904-b2d8-b69a962ca344-kube-api-access-9kr22\") pod \"payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8\" (UID: \"69e26174-f25d-4904-b2d8-b69a962ca344\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" Oct 30 12:42:19.579669 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:19.578337 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" Oct 30 12:42:19.828122 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:19.827920 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:42:19.829774 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:19.829371 4468 patch_prober.go:29] interesting pod/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 12:42:19.829774 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:19.829455 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" podUID="db33d91a-07fb-4c52-832e-0440bc4654af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 12:42:19.980178 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:19.980142 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8"] Oct 30 12:42:20.290164 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:20.270527 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" event={"ID":"69e26174-f25d-4904-b2d8-b69a962ca344","Type":"ContainerStarted","Data":"bc69f25f39073c9797f586e24247e937137594b2d4e1ff4b2170d8a9ee0c9cfd"} Oct 30 12:42:20.290995 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:20.290940 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" Oct 30 12:42:20.291181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:20.291155 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" event={"ID":"69e26174-f25d-4904-b2d8-b69a962ca344","Type":"ContainerStarted","Data":"57765afcb0954b25c4d13e9168e6539d4b0f7624c45aa577af44d8f36c8b4caa"} Oct 30 12:42:20.331176 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:20.324029 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" podStartSLOduration=1.324006277 podStartE2EDuration="1.324006277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:20.321517519 +0000 UTC m=+2891.355655916" watchObservedRunningTime="2026-10-30 12:42:20.324006277 +0000 UTC m=+2891.358144673" Oct 30 12:42:20.818098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:20.814460 4468 patch_prober.go:29] interesting pod/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 12:42:20.818098 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:20.814525 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" podUID="db33d91a-07fb-4c52-832e-0440bc4654af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 12:42:21.830093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:21.829297 4468 patch_prober.go:29] interesting pod/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 12:42:21.830093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:21.829362 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" podUID="db33d91a-07fb-4c52-832e-0440bc4654af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 12:42:22.817093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:22.810543 4468 patch_prober.go:29] interesting pod/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 12:42:22.817093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:22.810615 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" podUID="db33d91a-07fb-4c52-832e-0440bc4654af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 12:42:23.435210 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:23.435177 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:42:23.630398 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:42:23.629868 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:42:23.829147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:23.826547 4468 patch_prober.go:29] interesting pod/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 12:42:23.829147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:23.826613 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" podUID="db33d91a-07fb-4c52-832e-0440bc4654af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 12:42:24.813911 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:24.813875 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:42:24.815258 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:24.815235 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 12:42:26.512491 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:26.512457 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk"] Oct 30 12:42:26.513000 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:26.512701 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" podUID="51b18829-f11f-406e-99b0-decf56caaf67" containerName="istio-proxy" containerID="cri-o://6c568aaabbeecaf1aeb70fe46d455e5a96a406f86f51a3f0a9521a01f147cdf2" gracePeriod=30 Oct 30 12:42:29.781908 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:29.781865 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv"] Oct 30 12:42:29.787995 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:29.787964 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" Oct 30 12:42:29.805641 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:29.805609 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv"] Oct 30 12:42:29.812236 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:29.812200 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-12fd33\"" Oct 30 12:42:29.834141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:29.834090 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d45kt\" (UniqueName: \"kubernetes.io/projected/110bd521-953f-48c7-990a-823b164a281b-kube-api-access-d45kt\") pod \"payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv\" (UID: \"110bd521-953f-48c7-990a-823b164a281b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" Oct 30 12:42:29.834366 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:29.834208 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/110bd521-953f-48c7-990a-823b164a281b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv\" (UID: \"110bd521-953f-48c7-990a-823b164a281b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" Oct 30 12:42:30.031143 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:30.018700 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d45kt\" (UniqueName: \"kubernetes.io/projected/110bd521-953f-48c7-990a-823b164a281b-kube-api-access-d45kt\") pod \"payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv\" (UID: \"110bd521-953f-48c7-990a-823b164a281b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" Oct 30 12:42:30.090525 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:30.090432 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/110bd521-953f-48c7-990a-823b164a281b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv\" (UID: \"110bd521-953f-48c7-990a-823b164a281b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" Oct 30 12:42:30.094146 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:30.091767 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/110bd521-953f-48c7-990a-823b164a281b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv\" (UID: \"110bd521-953f-48c7-990a-823b164a281b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" Oct 30 12:42:30.094146 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:30.082003 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d45kt\" (UniqueName: \"kubernetes.io/projected/110bd521-953f-48c7-990a-823b164a281b-kube-api-access-d45kt\") pod \"payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv\" (UID: \"110bd521-953f-48c7-990a-823b164a281b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" Oct 30 12:42:30.116968 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:30.116907 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" Oct 30 12:42:30.458925 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:42:30.458888 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110bd521_953f_48c7_990a_823b164a281b.slice/crio-9241be5304741dc6c396aded48ebff2b46dc698ad345291e809e23100a1df438 WatchSource:0}: Error finding container 9241be5304741dc6c396aded48ebff2b46dc698ad345291e809e23100a1df438: Status 404 returned error can't find the container with id 9241be5304741dc6c396aded48ebff2b46dc698ad345291e809e23100a1df438 Oct 30 12:42:30.459709 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:30.459583 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv"] Oct 30 12:42:31.409670 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.409629 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" event={"ID":"110bd521-953f-48c7-990a-823b164a281b","Type":"ContainerStarted","Data":"22e689d84524c87a0d40c8b99895125f26a3aa09d97442cf9b2900e291ae7dc5"} Oct 30 12:42:31.410200 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.409693 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" event={"ID":"110bd521-953f-48c7-990a-823b164a281b","Type":"ContainerStarted","Data":"9241be5304741dc6c396aded48ebff2b46dc698ad345291e809e23100a1df438"} Oct 30 12:42:31.410200 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.409892 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" Oct 30 12:42:31.456147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.456082 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" podStartSLOduration=2.456054914 podStartE2EDuration="2.456054914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:31.454437083 +0000 UTC m=+2902.488575480" watchObservedRunningTime="2026-10-30 12:42:31.456054914 +0000 UTC m=+2902.490193309" Oct 30 12:42:31.803083 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.803043 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:42:31.926172 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.926117 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b18829-f11f-406e-99b0-decf56caaf67-istio-token\" (UniqueName: \"kubernetes.io/projected/51b18829-f11f-406e-99b0-decf56caaf67-istio-token\") pod \"51b18829-f11f-406e-99b0-decf56caaf67\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " Oct 30 12:42:31.926386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.926193 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b18829-f11f-406e-99b0-decf56caaf67-kube-api-access-tn2gq\" (UniqueName: \"kubernetes.io/projected/51b18829-f11f-406e-99b0-decf56caaf67-kube-api-access-tn2gq\") pod \"51b18829-f11f-406e-99b0-decf56caaf67\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " Oct 30 12:42:31.926386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.926266 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-istio-data\") pod \"51b18829-f11f-406e-99b0-decf56caaf67\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " Oct 30 12:42:31.926386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.926350 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-credential-socket\") pod \"51b18829-f11f-406e-99b0-decf56caaf67\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " Oct 30 12:42:31.926553 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.926385 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51b18829-f11f-406e-99b0-decf56caaf67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b18829-f11f-406e-99b0-decf56caaf67-istiod-ca-cert\") pod \"51b18829-f11f-406e-99b0-decf56caaf67\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " Oct 30 12:42:31.926553 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.926473 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-workload-socket\") pod \"51b18829-f11f-406e-99b0-decf56caaf67\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " Oct 30 12:42:31.926669 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.926603 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-workload-certs\") pod \"51b18829-f11f-406e-99b0-decf56caaf67\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " Oct 30 12:42:31.926669 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.926600 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-istio-data" pod "51b18829-f11f-406e-99b0-decf56caaf67" (UID: "51b18829-f11f-406e-99b0-decf56caaf67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:42:31.926669 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.926627 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-credential-socket" pod "51b18829-f11f-406e-99b0-decf56caaf67" (UID: "51b18829-f11f-406e-99b0-decf56caaf67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:42:31.926899 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.926717 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51b18829-f11f-406e-99b0-decf56caaf67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b18829-f11f-406e-99b0-decf56caaf67-istio-podinfo\") pod \"51b18829-f11f-406e-99b0-decf56caaf67\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " Oct 30 12:42:31.926899 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.926761 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-istio-envoy\") pod \"51b18829-f11f-406e-99b0-decf56caaf67\" (UID: \"51b18829-f11f-406e-99b0-decf56caaf67\") " Oct 30 12:42:31.926899 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.926752 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-workload-socket" pod "51b18829-f11f-406e-99b0-decf56caaf67" (UID: "51b18829-f11f-406e-99b0-decf56caaf67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:42:31.927032 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.926840 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b18829-f11f-406e-99b0-decf56caaf67-istiod-ca-cert" pod "51b18829-f11f-406e-99b0-decf56caaf67" (UID: "51b18829-f11f-406e-99b0-decf56caaf67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:42:31.927032 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.926873 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-workload-certs" pod "51b18829-f11f-406e-99b0-decf56caaf67" (UID: "51b18829-f11f-406e-99b0-decf56caaf67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:42:31.927505 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.927481 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:42:31.927663 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.927511 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:42:31.927663 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.927524 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:42:31.927663 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.927539 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b18829-f11f-406e-99b0-decf56caaf67-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:42:31.927663 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.927552 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:42:31.928601 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.928572 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b18829-f11f-406e-99b0-decf56caaf67-kube-api-access-tn2gq" pod "51b18829-f11f-406e-99b0-decf56caaf67" (UID: "51b18829-f11f-406e-99b0-decf56caaf67"). InnerVolumeSpecName "kube-api-access-tn2gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:42:31.928780 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.928669 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b18829-f11f-406e-99b0-decf56caaf67-istio-token" pod "51b18829-f11f-406e-99b0-decf56caaf67" (UID: "51b18829-f11f-406e-99b0-decf56caaf67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:42:31.929111 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.929061 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-istio-envoy" pod "51b18829-f11f-406e-99b0-decf56caaf67" (UID: "51b18829-f11f-406e-99b0-decf56caaf67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:42:31.929292 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:31.929271 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51b18829-f11f-406e-99b0-decf56caaf67-istio-podinfo" pod "51b18829-f11f-406e-99b0-decf56caaf67" (UID: "51b18829-f11f-406e-99b0-decf56caaf67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:42:32.029344 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.029239 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b18829-f11f-406e-99b0-decf56caaf67-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:42:32.029344 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.029276 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b18829-f11f-406e-99b0-decf56caaf67-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:42:32.029344 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.029289 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51b18829-f11f-406e-99b0-decf56caaf67-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:42:32.029344 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.029300 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn2gq\" (UniqueName: \"kubernetes.io/projected/51b18829-f11f-406e-99b0-decf56caaf67-kube-api-access-tn2gq\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:42:32.197008 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.196962 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg"] Oct 30 12:42:32.197850 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.197829 4468 cpu_manager.go:414] "containerMap: removing container" podUID="51b18829-f11f-406e-99b0-decf56caaf67" containerName="istio-proxy" Oct 30 12:42:32.197908 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.197851 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b18829-f11f-406e-99b0-decf56caaf67" containerName="istio-proxy" Oct 30 12:42:32.197962 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.197953 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b18829-f11f-406e-99b0-decf56caaf67" containerName="istio-proxy" Oct 30 12:42:32.202003 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.201977 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" Oct 30 12:42:32.218239 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.218205 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0de4b9\"" Oct 30 12:42:32.219270 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.219245 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0de4b9-dockercfg-jmbk5\"" Oct 30 12:42:32.221871 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.221836 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg"] Oct 30 12:42:32.346161 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.345139 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9"] Oct 30 12:42:32.385432 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.385393 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9"] Oct 30 12:42:32.385649 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.385535 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" Oct 30 12:42:32.391384 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.391353 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" Oct 30 12:42:32.407523 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.399054 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qd46\" (UniqueName: \"kubernetes.io/projected/065b66a4-c80e-4dde-8bf6-4f14386e6d87-kube-api-access-5qd46\") pod \"payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg\" (UID: \"065b66a4-c80e-4dde-8bf6-4f14386e6d87\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" Oct 30 12:42:32.407523 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.399326 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065b66a4-c80e-4dde-8bf6-4f14386e6d87-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg\" (UID: \"065b66a4-c80e-4dde-8bf6-4f14386e6d87\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" Oct 30 12:42:32.428423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.425606 4468 generic.go:356] "Generic (PLEG): container finished" podID="51b18829-f11f-406e-99b0-decf56caaf67" containerID="6c568aaabbeecaf1aeb70fe46d455e5a96a406f86f51a3f0a9521a01f147cdf2" exitCode=0 Oct 30 12:42:32.428423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.426400 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" Oct 30 12:42:32.428423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.427061 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" event={"ID":"51b18829-f11f-406e-99b0-decf56caaf67","Type":"ContainerDied","Data":"6c568aaabbeecaf1aeb70fe46d455e5a96a406f86f51a3f0a9521a01f147cdf2"} Oct 30 12:42:32.428423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.427165 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk" event={"ID":"51b18829-f11f-406e-99b0-decf56caaf67","Type":"ContainerDied","Data":"95551f310c000b465c6faf564740d29e02575c63d3769e2dfd48df12b34fee95"} Oct 30 12:42:32.428423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.427223 4468 scope.go:122] "RemoveContainer" containerID="6c568aaabbeecaf1aeb70fe46d455e5a96a406f86f51a3f0a9521a01f147cdf2" Oct 30 12:42:32.443131 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.443048 4468 scope.go:122] "RemoveContainer" containerID="6c568aaabbeecaf1aeb70fe46d455e5a96a406f86f51a3f0a9521a01f147cdf2" Oct 30 12:42:32.443486 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:42:32.443463 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c568aaabbeecaf1aeb70fe46d455e5a96a406f86f51a3f0a9521a01f147cdf2\": container with ID starting with 6c568aaabbeecaf1aeb70fe46d455e5a96a406f86f51a3f0a9521a01f147cdf2 not found: ID does not exist" containerID="6c568aaabbeecaf1aeb70fe46d455e5a96a406f86f51a3f0a9521a01f147cdf2" Oct 30 12:42:32.443560 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.443496 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c568aaabbeecaf1aeb70fe46d455e5a96a406f86f51a3f0a9521a01f147cdf2"} err="failed to get container status \"6c568aaabbeecaf1aeb70fe46d455e5a96a406f86f51a3f0a9521a01f147cdf2\": rpc error: code = NotFound desc = could not find container \"6c568aaabbeecaf1aeb70fe46d455e5a96a406f86f51a3f0a9521a01f147cdf2\": container with ID starting with 6c568aaabbeecaf1aeb70fe46d455e5a96a406f86f51a3f0a9521a01f147cdf2 not found: ID does not exist" Oct 30 12:42:32.501158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.500873 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qd46\" (UniqueName: \"kubernetes.io/projected/065b66a4-c80e-4dde-8bf6-4f14386e6d87-kube-api-access-5qd46\") pod \"payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg\" (UID: \"065b66a4-c80e-4dde-8bf6-4f14386e6d87\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" Oct 30 12:42:32.501158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.501030 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab60f224-4948-4a2b-abf2-116cb1827625-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9\" (UID: \"ab60f224-4948-4a2b-abf2-116cb1827625\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" Oct 30 12:42:32.501158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.501116 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4rnz\" (UniqueName: \"kubernetes.io/projected/ab60f224-4948-4a2b-abf2-116cb1827625-kube-api-access-h4rnz\") pod \"payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9\" (UID: \"ab60f224-4948-4a2b-abf2-116cb1827625\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" Oct 30 12:42:32.502240 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.501211 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065b66a4-c80e-4dde-8bf6-4f14386e6d87-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg\" (UID: \"065b66a4-c80e-4dde-8bf6-4f14386e6d87\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" Oct 30 12:42:32.502324 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.502304 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065b66a4-c80e-4dde-8bf6-4f14386e6d87-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg\" (UID: \"065b66a4-c80e-4dde-8bf6-4f14386e6d87\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" Oct 30 12:42:32.526021 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.525985 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qd46\" (UniqueName: \"kubernetes.io/projected/065b66a4-c80e-4dde-8bf6-4f14386e6d87-kube-api-access-5qd46\") pod \"payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg\" (UID: \"065b66a4-c80e-4dde-8bf6-4f14386e6d87\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" Oct 30 12:42:32.546850 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.546803 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk"] Oct 30 12:42:32.552346 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.552306 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-478gk"] Oct 30 12:42:32.602951 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.602846 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab60f224-4948-4a2b-abf2-116cb1827625-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9\" (UID: \"ab60f224-4948-4a2b-abf2-116cb1827625\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" Oct 30 12:42:32.602951 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.602905 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4rnz\" (UniqueName: \"kubernetes.io/projected/ab60f224-4948-4a2b-abf2-116cb1827625-kube-api-access-h4rnz\") pod \"payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9\" (UID: \"ab60f224-4948-4a2b-abf2-116cb1827625\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" Oct 30 12:42:32.603653 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.603631 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab60f224-4948-4a2b-abf2-116cb1827625-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9\" (UID: \"ab60f224-4948-4a2b-abf2-116cb1827625\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" Oct 30 12:42:32.627803 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.627770 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4rnz\" (UniqueName: \"kubernetes.io/projected/ab60f224-4948-4a2b-abf2-116cb1827625-kube-api-access-h4rnz\") pod \"payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9\" (UID: \"ab60f224-4948-4a2b-abf2-116cb1827625\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" Oct 30 12:42:32.733720 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.733686 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" Oct 30 12:42:32.818720 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:32.818671 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" Oct 30 12:42:33.040960 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:42:33.040928 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab60f224_4948_4a2b_abf2_116cb1827625.slice/crio-1a29882f1cb29e77d78fe87df13b0d5b63dcb1eef7158b180ee91760f61d423d WatchSource:0}: Error finding container 1a29882f1cb29e77d78fe87df13b0d5b63dcb1eef7158b180ee91760f61d423d: Status 404 returned error can't find the container with id 1a29882f1cb29e77d78fe87df13b0d5b63dcb1eef7158b180ee91760f61d423d Oct 30 12:42:33.044234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:33.044204 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9"] Oct 30 12:42:33.093411 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:33.093376 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51b18829-f11f-406e-99b0-decf56caaf67" path="/var/lib/kubelet/pods/51b18829-f11f-406e-99b0-decf56caaf67/volumes" Oct 30 12:42:33.142559 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:42:33.142472 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065b66a4_c80e_4dde_8bf6_4f14386e6d87.slice/crio-827bca12bbb11531a1bf2d737cb746d50570047f070f5a73d252d1bdc37f24a8 WatchSource:0}: Error finding container 827bca12bbb11531a1bf2d737cb746d50570047f070f5a73d252d1bdc37f24a8: Status 404 returned error can't find the container with id 827bca12bbb11531a1bf2d737cb746d50570047f070f5a73d252d1bdc37f24a8 Oct 30 12:42:33.144807 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:33.144722 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg"] Oct 30 12:42:33.434594 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:33.434553 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" event={"ID":"ab60f224-4948-4a2b-abf2-116cb1827625","Type":"ContainerStarted","Data":"0a58c2de2b8acde39654c3f72789c720e25918f63388c024e6a5ba4eb9f19354"} Oct 30 12:42:33.435095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:33.434624 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" event={"ID":"ab60f224-4948-4a2b-abf2-116cb1827625","Type":"ContainerStarted","Data":"1a29882f1cb29e77d78fe87df13b0d5b63dcb1eef7158b180ee91760f61d423d"} Oct 30 12:42:33.435095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:33.434765 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" Oct 30 12:42:33.436954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:33.436925 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" event={"ID":"065b66a4-c80e-4dde-8bf6-4f14386e6d87","Type":"ContainerStarted","Data":"8da48b357841d8f1ea105eb6ccb75dafc58aa2b91555ebdea60ce07082755439"} Oct 30 12:42:33.437093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:33.436970 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" event={"ID":"065b66a4-c80e-4dde-8bf6-4f14386e6d87","Type":"ContainerStarted","Data":"827bca12bbb11531a1bf2d737cb746d50570047f070f5a73d252d1bdc37f24a8"} Oct 30 12:42:33.437093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:33.437056 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" Oct 30 12:42:33.480215 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:33.480155 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" podStartSLOduration=1.480139503 podStartE2EDuration="1.480139503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:33.478136432 +0000 UTC m=+2904.512274828" watchObservedRunningTime="2026-10-30 12:42:33.480139503 +0000 UTC m=+2904.514277899" Oct 30 12:42:33.525221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:33.525166 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" podStartSLOduration=1.525149543 podStartE2EDuration="1.525149543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:33.522726729 +0000 UTC m=+2904.556865124" watchObservedRunningTime="2026-10-30 12:42:33.525149543 +0000 UTC m=+2904.559287939" Oct 30 12:42:42.430130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:42.430065 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" Oct 30 12:42:44.447340 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:44.447307 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" Oct 30 12:42:44.447836 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:44.447401 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" Oct 30 12:42:53.188524 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:53.188433 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb"] Oct 30 12:42:53.193788 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:53.193762 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" Oct 30 12:42:53.217624 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:53.217587 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb"] Oct 30 12:42:53.223752 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:53.223716 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-0f7d14-serving-cert\"" Oct 30 12:42:53.248641 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:53.248603 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af770c09-3be0-418b-8737-3cd556b08035-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb\" (UID: \"af770c09-3be0-418b-8737-3cd556b08035\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" Oct 30 12:42:53.248825 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:53.248668 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af770c09-3be0-418b-8737-3cd556b08035-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb\" (UID: \"af770c09-3be0-418b-8737-3cd556b08035\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" Oct 30 12:42:53.248825 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:53.248701 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af770c09-3be0-418b-8737-3cd556b08035-metrics-certs\") pod \"maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb\" (UID: \"af770c09-3be0-418b-8737-3cd556b08035\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" Oct 30 12:42:53.248825 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:53.248750 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n59p\" (UniqueName: \"kubernetes.io/projected/af770c09-3be0-418b-8737-3cd556b08035-kube-api-access-2n59p\") pod \"maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb\" (UID: \"af770c09-3be0-418b-8737-3cd556b08035\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" Oct 30 12:42:53.349606 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:53.349560 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af770c09-3be0-418b-8737-3cd556b08035-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb\" (UID: \"af770c09-3be0-418b-8737-3cd556b08035\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" Oct 30 12:42:53.349796 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:53.349615 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af770c09-3be0-418b-8737-3cd556b08035-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb\" (UID: \"af770c09-3be0-418b-8737-3cd556b08035\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" Oct 30 12:42:53.349796 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:53.349636 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af770c09-3be0-418b-8737-3cd556b08035-metrics-certs\") pod \"maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb\" (UID: \"af770c09-3be0-418b-8737-3cd556b08035\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" Oct 30 12:42:53.349796 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:53.349664 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n59p\" (UniqueName: \"kubernetes.io/projected/af770c09-3be0-418b-8737-3cd556b08035-kube-api-access-2n59p\") pod \"maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb\" (UID: \"af770c09-3be0-418b-8737-3cd556b08035\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" Oct 30 12:42:53.349796 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:42:53.349761 4468 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-serving-cert: secret "maas-api-e2e-extmod-b-0f7d14-serving-cert" not found Oct 30 12:42:53.350051 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:42:53.349850 4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af770c09-3be0-418b-8737-3cd556b08035-maas-api-tls podName:af770c09-3be0-418b-8737-3cd556b08035 nodeName:}" failed. No retries permitted until 2026-10-30 12:42:53.849829235 +0000 UTC m=+2924.883967624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af770c09-3be0-418b-8737-3cd556b08035-maas-api-tls") pod "maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" (UID: "af770c09-3be0-418b-8737-3cd556b08035") : secret "maas-api-e2e-extmod-b-0f7d14-serving-cert" not found Oct 30 12:42:53.350717 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:53.350685 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af770c09-3be0-418b-8737-3cd556b08035-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb\" (UID: \"af770c09-3be0-418b-8737-3cd556b08035\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" Oct 30 12:42:53.352621 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:53.352588 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af770c09-3be0-418b-8737-3cd556b08035-metrics-certs\") pod \"maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb\" (UID: \"af770c09-3be0-418b-8737-3cd556b08035\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" Oct 30 12:42:53.411095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:53.407462 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n59p\" (UniqueName: \"kubernetes.io/projected/af770c09-3be0-418b-8737-3cd556b08035-kube-api-access-2n59p\") pod \"maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb\" (UID: \"af770c09-3be0-418b-8737-3cd556b08035\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" Oct 30 12:42:53.609412 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:42:53.609355 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:42:53.857493 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:53.857409 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af770c09-3be0-418b-8737-3cd556b08035-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb\" (UID: \"af770c09-3be0-418b-8737-3cd556b08035\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" Oct 30 12:42:53.860000 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:53.859959 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af770c09-3be0-418b-8737-3cd556b08035-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb\" (UID: \"af770c09-3be0-418b-8737-3cd556b08035\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" Oct 30 12:42:54.114290 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:54.114188 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" Oct 30 12:42:54.515959 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:54.515909 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb"] Oct 30 12:42:54.644188 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:54.644153 4468 generic.go:356] "Generic (PLEG): container finished" podID="0b15b44d-dfe6-4001-966b-f72c56f449c1" containerID="c26876500b1851d4d59a53365d3ee95f0d5a608feb2148d066743c2686108c5c" exitCode=0 Oct 30 12:42:54.644390 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:54.644233 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" event={"ID":"0b15b44d-dfe6-4001-966b-f72c56f449c1","Type":"ContainerDied","Data":"c26876500b1851d4d59a53365d3ee95f0d5a608feb2148d066743c2686108c5c"} Oct 30 12:42:54.644763 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:54.644742 4468 scope.go:122] "RemoveContainer" containerID="c26876500b1851d4d59a53365d3ee95f0d5a608feb2148d066743c2686108c5c" Oct 30 12:42:54.645687 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:54.645658 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" event={"ID":"af770c09-3be0-418b-8737-3cd556b08035","Type":"ContainerStarted","Data":"e6b0818128dc3fd464a264423f33ef36951873ceac406d238ee01ff0d4f31712"} Oct 30 12:42:54.646671 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:54.646643 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:54.646737 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:54.646718 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:54.646775 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:54.646749 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:42:55.656742 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:55.656708 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" event={"ID":"0b15b44d-dfe6-4001-966b-f72c56f449c1","Type":"ContainerStarted","Data":"1e446782f77f57beb977f27cf067f78272711ef1b394eff8084889ea914d650b"} Oct 30 12:42:55.658539 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:55.658498 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" event={"ID":"af770c09-3be0-418b-8737-3cd556b08035","Type":"ContainerStarted","Data":"fca61f1217195aa778e20fec43b044864ed365b94fd28e920623988393f4ee93"} Oct 30 12:42:55.658675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:55.658585 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" Oct 30 12:42:55.684117 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:55.684059 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:42:55.684373 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:55.684347 4468 patch_prober.go:29] interesting pod/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 12:42:55.684458 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:55.684415 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" podUID="0b15b44d-dfe6-4001-966b-f72c56f449c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 12:42:55.787473 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:55.787399 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" podStartSLOduration=2.298317867 podStartE2EDuration="2.787381546s" totalImagesPullingTime="489.063679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:55.783216281 +0000 UTC m=+2926.817354676" watchObservedRunningTime="2026-10-30 12:42:55.787381546 +0000 UTC m=+2926.821519942" Oct 30 12:42:56.684886 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:56.684847 4468 patch_prober.go:29] interesting pod/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 12:42:56.685395 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:56.684930 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" podUID="0b15b44d-dfe6-4001-966b-f72c56f449c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 12:42:57.280119 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:57.280061 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq"] Oct 30 12:42:57.284990 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:57.284954 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq"] Oct 30 12:42:57.285206 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:57.285103 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" Oct 30 12:42:57.302701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:57.302667 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-ecb3aa\"" Oct 30 12:42:57.318415 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:57.318372 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-ecb3aa-dockercfg-5mf57\"" Oct 30 12:42:57.429161 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:57.429063 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba1e2c95-cd0f-43dc-89be-607ea62fd51d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq\" (UID: \"ba1e2c95-cd0f-43dc-89be-607ea62fd51d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" Oct 30 12:42:57.429389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:57.429269 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqpr6\" (UniqueName: \"kubernetes.io/projected/ba1e2c95-cd0f-43dc-89be-607ea62fd51d-kube-api-access-bqpr6\") pod \"payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq\" (UID: \"ba1e2c95-cd0f-43dc-89be-607ea62fd51d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" Oct 30 12:42:57.530390 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:57.530287 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba1e2c95-cd0f-43dc-89be-607ea62fd51d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq\" (UID: \"ba1e2c95-cd0f-43dc-89be-607ea62fd51d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" Oct 30 12:42:57.530586 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:57.530409 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqpr6\" (UniqueName: \"kubernetes.io/projected/ba1e2c95-cd0f-43dc-89be-607ea62fd51d-kube-api-access-bqpr6\") pod \"payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq\" (UID: \"ba1e2c95-cd0f-43dc-89be-607ea62fd51d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" Oct 30 12:42:57.531289 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:57.531264 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba1e2c95-cd0f-43dc-89be-607ea62fd51d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq\" (UID: \"ba1e2c95-cd0f-43dc-89be-607ea62fd51d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" Oct 30 12:42:57.572509 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:57.572474 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqpr6\" (UniqueName: \"kubernetes.io/projected/ba1e2c95-cd0f-43dc-89be-607ea62fd51d-kube-api-access-bqpr6\") pod \"payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq\" (UID: \"ba1e2c95-cd0f-43dc-89be-607ea62fd51d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" Oct 30 12:42:57.618105 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:57.610602 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" Oct 30 12:42:57.684329 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:57.684298 4468 patch_prober.go:29] interesting pod/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 12:42:57.684476 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:57.684363 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" podUID="0b15b44d-dfe6-4001-966b-f72c56f449c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 12:42:57.939926 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:42:57.939886 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1e2c95_cd0f_43dc_89be_607ea62fd51d.slice/crio-8568aabad39c3b0ed7329f79588d246ea330f8ded646557138905157f50cf0b1 WatchSource:0}: Error finding container 8568aabad39c3b0ed7329f79588d246ea330f8ded646557138905157f50cf0b1: Status 404 returned error can't find the container with id 8568aabad39c3b0ed7329f79588d246ea330f8ded646557138905157f50cf0b1 Oct 30 12:42:57.943254 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:57.943222 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq"] Oct 30 12:42:58.684021 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:58.683989 4468 patch_prober.go:29] interesting pod/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 12:42:58.684256 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:58.684040 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" podUID="0b15b44d-dfe6-4001-966b-f72c56f449c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 12:42:58.684676 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:58.684652 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" event={"ID":"ba1e2c95-cd0f-43dc-89be-607ea62fd51d","Type":"ContainerStarted","Data":"50bf3c9977b80822fb7d885e5be7ac2584302ced6f4b98ef1f7bce05dac6dfc0"} Oct 30 12:42:58.684801 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:58.684698 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" event={"ID":"ba1e2c95-cd0f-43dc-89be-607ea62fd51d","Type":"ContainerStarted","Data":"8568aabad39c3b0ed7329f79588d246ea330f8ded646557138905157f50cf0b1"} Oct 30 12:42:58.684801 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:58.684732 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" Oct 30 12:42:59.684468 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:59.684437 4468 patch_prober.go:29] interesting pod/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 12:42:59.684866 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:42:59.684490 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" podUID="0b15b44d-dfe6-4001-966b-f72c56f449c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 12:43:00.684946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:00.684908 4468 patch_prober.go:29] interesting pod/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 12:43:00.685405 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:00.684965 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" podUID="0b15b44d-dfe6-4001-966b-f72c56f449c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 12:43:01.632119 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:01.632052 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:43:01.673681 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:01.673648 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" Oct 30 12:43:01.689152 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:01.689123 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:43:01.726314 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:01.726255 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" podStartSLOduration=4.726237695 podStartE2EDuration="4.726237695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:58.736041162 +0000 UTC m=+2929.770179559" watchObservedRunningTime="2026-10-30 12:43:01.726237695 +0000 UTC m=+2932.760376090" Oct 30 12:43:01.740767 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:01.740738 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 12:43:08.446103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:08.446047 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm"] Oct 30 12:43:08.450798 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:08.450773 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" Oct 30 12:43:08.471883 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:08.471851 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm"] Oct 30 12:43:08.474738 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:08.474709 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-12fd33-serving-cert\"" Oct 30 12:43:08.479757 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:08.479668 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a880cac8-149c-42bc-87c1-31464bf24279-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm\" (UID: \"a880cac8-149c-42bc-87c1-31464bf24279\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" Oct 30 12:43:08.480158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:08.480125 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a880cac8-149c-42bc-87c1-31464bf24279-maas-api-tls\") pod \"maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm\" (UID: \"a880cac8-149c-42bc-87c1-31464bf24279\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" Oct 30 12:43:08.480280 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:08.480203 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m82k\" (UniqueName: \"kubernetes.io/projected/a880cac8-149c-42bc-87c1-31464bf24279-kube-api-access-6m82k\") pod \"maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm\" (UID: \"a880cac8-149c-42bc-87c1-31464bf24279\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" Oct 30 12:43:08.480280 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:08.480233 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a880cac8-149c-42bc-87c1-31464bf24279-metrics-certs\") pod \"maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm\" (UID: \"a880cac8-149c-42bc-87c1-31464bf24279\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" Oct 30 12:43:08.582037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:08.581998 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a880cac8-149c-42bc-87c1-31464bf24279-maas-api-tls\") pod \"maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm\" (UID: \"a880cac8-149c-42bc-87c1-31464bf24279\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" Oct 30 12:43:08.582273 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:08.582056 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m82k\" (UniqueName: \"kubernetes.io/projected/a880cac8-149c-42bc-87c1-31464bf24279-kube-api-access-6m82k\") pod \"maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm\" (UID: \"a880cac8-149c-42bc-87c1-31464bf24279\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" Oct 30 12:43:08.582273 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:08.582090 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a880cac8-149c-42bc-87c1-31464bf24279-metrics-certs\") pod \"maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm\" (UID: \"a880cac8-149c-42bc-87c1-31464bf24279\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" Oct 30 12:43:08.582273 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:08.582198 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a880cac8-149c-42bc-87c1-31464bf24279-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm\" (UID: \"a880cac8-149c-42bc-87c1-31464bf24279\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" Oct 30 12:43:08.582892 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:08.582862 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a880cac8-149c-42bc-87c1-31464bf24279-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm\" (UID: \"a880cac8-149c-42bc-87c1-31464bf24279\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" Oct 30 12:43:08.584740 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:08.584719 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a880cac8-149c-42bc-87c1-31464bf24279-maas-api-tls\") pod \"maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm\" (UID: \"a880cac8-149c-42bc-87c1-31464bf24279\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" Oct 30 12:43:08.584828 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:08.584764 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a880cac8-149c-42bc-87c1-31464bf24279-metrics-certs\") pod \"maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm\" (UID: \"a880cac8-149c-42bc-87c1-31464bf24279\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" Oct 30 12:43:08.610213 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:08.610183 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m82k\" (UniqueName: \"kubernetes.io/projected/a880cac8-149c-42bc-87c1-31464bf24279-kube-api-access-6m82k\") pod \"maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm\" (UID: \"a880cac8-149c-42bc-87c1-31464bf24279\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" Oct 30 12:43:08.768944 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:08.768845 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" Oct 30 12:43:09.102342 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:43:09.102312 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda880cac8_149c_42bc_87c1_31464bf24279.slice/crio-a6a1f382205360884dc896eb50d1daf94413cba53d217de13026e0d41822475a WatchSource:0}: Error finding container a6a1f382205360884dc896eb50d1daf94413cba53d217de13026e0d41822475a: Status 404 returned error can't find the container with id a6a1f382205360884dc896eb50d1daf94413cba53d217de13026e0d41822475a Oct 30 12:43:09.105907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:09.105883 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm"] Oct 30 12:43:09.711964 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:09.710608 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" Oct 30 12:43:09.813969 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:09.813906 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" event={"ID":"a880cac8-149c-42bc-87c1-31464bf24279","Type":"ContainerStarted","Data":"824134fccb9e1550302b953bc9637d7d1fb18cd72e70ef2138a7da4e4223bf55"} Oct 30 12:43:09.814171 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:09.813986 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" event={"ID":"a880cac8-149c-42bc-87c1-31464bf24279","Type":"ContainerStarted","Data":"a6a1f382205360884dc896eb50d1daf94413cba53d217de13026e0d41822475a"} Oct 30 12:43:09.814274 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:09.814247 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" Oct 30 12:43:09.878713 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:09.878641 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" podStartSLOduration=1.5392876960000001 podStartE2EDuration="1.878619518s" totalImagesPullingTime="339.331822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:43:09.874580429 +0000 UTC m=+2940.908718825" watchObservedRunningTime="2026-10-30 12:43:09.878619518 +0000 UTC m=+2940.912757915" Oct 30 12:43:15.856706 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:15.856031 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" Oct 30 12:43:23.601373 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:43:23.601332 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:43:46.401785 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:46.401749 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq"] Oct 30 12:43:51.402259 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:51.402201 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq" podUID="d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a" containerName="payload-processing" containerID="cri-o://22a5cbd77164e9d4e5c9a698770830fa1fb61f047806381c25b5c0436c91756f" gracePeriod=25 Oct 30 12:43:51.696436 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:51.696406 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq" Oct 30 12:43:51.750377 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:51.750340 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a-plugins-config-volume\") pod \"d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a\" (UID: \"d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a\") " Oct 30 12:43:51.750574 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:51.750539 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a-kube-api-access-m7tvp\" (UniqueName: \"kubernetes.io/projected/d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a-kube-api-access-m7tvp\") pod \"d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a\" (UID: \"d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a\") " Oct 30 12:43:51.750792 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:51.750763 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a-plugins-config-volume" pod "d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a" (UID: "d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:43:51.751366 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:51.751338 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:43:51.753166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:51.753141 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a-kube-api-access-m7tvp" pod "d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a" (UID: "d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a"). InnerVolumeSpecName "kube-api-access-m7tvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:51.852559 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:51.852515 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7tvp\" (UniqueName: \"kubernetes.io/projected/d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a-kube-api-access-m7tvp\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:43:52.284432 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:52.284394 4468 generic.go:356] "Generic (PLEG): container finished" podID="d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a" containerID="22a5cbd77164e9d4e5c9a698770830fa1fb61f047806381c25b5c0436c91756f" exitCode=0 Oct 30 12:43:52.284617 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:52.284449 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq" Oct 30 12:43:52.284617 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:52.284464 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq" event={"ID":"d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a","Type":"ContainerDied","Data":"22a5cbd77164e9d4e5c9a698770830fa1fb61f047806381c25b5c0436c91756f"} Oct 30 12:43:52.284617 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:52.284520 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq" event={"ID":"d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a","Type":"ContainerDied","Data":"11316a9787769b9cf24e0be75bd75bf7613e917d942d552efae70f804b23010b"} Oct 30 12:43:52.284617 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:52.284549 4468 scope.go:122] "RemoveContainer" containerID="22a5cbd77164e9d4e5c9a698770830fa1fb61f047806381c25b5c0436c91756f" Oct 30 12:43:52.295603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:52.295582 4468 scope.go:122] "RemoveContainer" containerID="22a5cbd77164e9d4e5c9a698770830fa1fb61f047806381c25b5c0436c91756f" Oct 30 12:43:52.295968 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:43:52.295941 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22a5cbd77164e9d4e5c9a698770830fa1fb61f047806381c25b5c0436c91756f\": container with ID starting with 22a5cbd77164e9d4e5c9a698770830fa1fb61f047806381c25b5c0436c91756f not found: ID does not exist" containerID="22a5cbd77164e9d4e5c9a698770830fa1fb61f047806381c25b5c0436c91756f" Oct 30 12:43:52.296027 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:52.295981 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22a5cbd77164e9d4e5c9a698770830fa1fb61f047806381c25b5c0436c91756f"} err="failed to get container status \"22a5cbd77164e9d4e5c9a698770830fa1fb61f047806381c25b5c0436c91756f\": rpc error: code = NotFound desc = could not find container \"22a5cbd77164e9d4e5c9a698770830fa1fb61f047806381c25b5c0436c91756f\": container with ID starting with 22a5cbd77164e9d4e5c9a698770830fa1fb61f047806381c25b5c0436c91756f not found: ID does not exist" Oct 30 12:43:52.345372 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:52.345339 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq"] Oct 30 12:43:52.357364 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:52.357329 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ac674b-7b6468ff8-xnqpq"] Oct 30 12:43:53.092053 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:43:53.092019 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a" path="/var/lib/kubelet/pods/d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a/volumes" Oct 30 12:43:53.600610 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:43:53.600575 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:44:05.924922 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:05.924869 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww"] Oct 30 12:44:05.925690 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:05.925673 4468 cpu_manager.go:414] "containerMap: removing container" podUID="d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a" containerName="payload-processing" Oct 30 12:44:05.925690 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:05.925690 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a" containerName="payload-processing" Oct 30 12:44:05.925823 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:05.925814 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d39dd7e1-64b2-4ec3-a7ea-c94ee9299f3a" containerName="payload-processing" Oct 30 12:44:05.931368 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:05.931342 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" Oct 30 12:44:05.968721 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:05.968682 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12fd33\"/\"kube-root-ca.crt\"" Oct 30 12:44:05.972639 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:05.969117 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12fd33\"/\"payload-processing-external-model-e2e--54fdd6e5-dockercfg-zzm2p\"" Oct 30 12:44:05.972905 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:05.969383 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12fd33\"/\"routing-overlay\"" Oct 30 12:44:05.973780 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:05.969436 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12fd33\"/\"payload-processing-external-model-plugins-e2e-extmod-b-12fd33\"" Oct 30 12:44:05.973946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:05.969518 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12fd33\"/\"openshift-service-ca.crt\"" Oct 30 12:44:05.974087 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:05.969573 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12fd33\"/\"agc-cred-de8df29d108f\"" Oct 30 12:44:05.974211 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:05.971117 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vffb9\" (UniqueName: \"kubernetes.io/projected/af776420-bc69-4979-9d77-ac5c7d22c2a4-kube-api-access-vffb9\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww\" (UID: \"af776420-bc69-4979-9d77-ac5c7d22c2a4\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" Oct 30 12:44:05.974406 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:05.974387 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af776420-bc69-4979-9d77-ac5c7d22c2a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww\" (UID: \"af776420-bc69-4979-9d77-ac5c7d22c2a4\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" Oct 30 12:44:05.974649 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:05.974632 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af776420-bc69-4979-9d77-ac5c7d22c2a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww\" (UID: \"af776420-bc69-4979-9d77-ac5c7d22c2a4\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" Oct 30 12:44:05.974860 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:05.974843 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af776420-bc69-4979-9d77-ac5c7d22c2a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww\" (UID: \"af776420-bc69-4979-9d77-ac5c7d22c2a4\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" Oct 30 12:44:06.104091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:06.081655 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vffb9\" (UniqueName: \"kubernetes.io/projected/af776420-bc69-4979-9d77-ac5c7d22c2a4-kube-api-access-vffb9\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww\" (UID: \"af776420-bc69-4979-9d77-ac5c7d22c2a4\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" Oct 30 12:44:06.104836 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:06.096922 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww"] Oct 30 12:44:06.105062 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:06.105041 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af776420-bc69-4979-9d77-ac5c7d22c2a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww\" (UID: \"af776420-bc69-4979-9d77-ac5c7d22c2a4\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" Oct 30 12:44:06.110313 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:06.110269 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af776420-bc69-4979-9d77-ac5c7d22c2a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww\" (UID: \"af776420-bc69-4979-9d77-ac5c7d22c2a4\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" Oct 30 12:44:06.110587 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:06.110563 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af776420-bc69-4979-9d77-ac5c7d22c2a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww\" (UID: \"af776420-bc69-4979-9d77-ac5c7d22c2a4\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" Oct 30 12:44:06.112220 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:06.106897 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af776420-bc69-4979-9d77-ac5c7d22c2a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww\" (UID: \"af776420-bc69-4979-9d77-ac5c7d22c2a4\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" Oct 30 12:44:06.112220 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:06.112174 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af776420-bc69-4979-9d77-ac5c7d22c2a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww\" (UID: \"af776420-bc69-4979-9d77-ac5c7d22c2a4\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" Oct 30 12:44:06.117829 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:06.117723 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af776420-bc69-4979-9d77-ac5c7d22c2a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww\" (UID: \"af776420-bc69-4979-9d77-ac5c7d22c2a4\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" Oct 30 12:44:06.120682 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:06.120623 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vffb9\" (UniqueName: \"kubernetes.io/projected/af776420-bc69-4979-9d77-ac5c7d22c2a4-kube-api-access-vffb9\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww\" (UID: \"af776420-bc69-4979-9d77-ac5c7d22c2a4\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" Oct 30 12:44:06.247716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:06.247629 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" Oct 30 12:44:06.573319 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:44:06.573289 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf776420_bc69_4979_9d77_ac5c7d22c2a4.slice/crio-2ec4c0b99e9b2aca22bd3ba5409cd71eb435facd52a45ab49bb495319aea6aba WatchSource:0}: Error finding container 2ec4c0b99e9b2aca22bd3ba5409cd71eb435facd52a45ab49bb495319aea6aba: Status 404 returned error can't find the container with id 2ec4c0b99e9b2aca22bd3ba5409cd71eb435facd52a45ab49bb495319aea6aba Oct 30 12:44:06.578586 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:06.578561 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww"] Oct 30 12:44:07.438121 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:07.438056 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" event={"ID":"af776420-bc69-4979-9d77-ac5c7d22c2a4","Type":"ContainerStarted","Data":"c65b09051e4fd66920578813d4c5797fa678628b0a5f65950e7f0cfbc80e2f8f"} Oct 30 12:44:07.438519 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:07.438148 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" event={"ID":"af776420-bc69-4979-9d77-ac5c7d22c2a4","Type":"ContainerStarted","Data":"2ec4c0b99e9b2aca22bd3ba5409cd71eb435facd52a45ab49bb495319aea6aba"} Oct 30 12:44:07.438519 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:07.438176 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" Oct 30 12:44:18.481451 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:18.481416 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" Oct 30 12:44:18.581617 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:44:18.581558 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" podStartSLOduration=13.58154142 podStartE2EDuration="13.58154142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:07.486579224 +0000 UTC m=+2998.520717620" watchObservedRunningTime="2026-10-30 12:44:18.58154142 +0000 UTC m=+3009.615679845" Oct 30 12:44:23.601931 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:44:23.601842 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:44:53.602750 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:44:53.602710 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:45:00.318583 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.318538 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb"] Oct 30 12:45:00.328514 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.328477 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" Oct 30 12:45:00.346928 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.346892 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb"] Oct 30 12:45:00.362892 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.362850 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:45:00.380351 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.380312 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f"] Oct 30 12:45:00.386596 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.386564 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" Oct 30 12:45:00.437461 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.437419 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f"] Oct 30 12:45:00.449114 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.449060 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d"] Oct 30 12:45:00.449378 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.449151 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-569vj\" (UniqueName: \"kubernetes.io/projected/aafad9d8-210e-4385-8d3a-447fa25f1d53-kube-api-access-569vj\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f\" (UID: \"aafad9d8-210e-4385-8d3a-447fa25f1d53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" Oct 30 12:45:00.449378 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.449194 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7klw8\" (UniqueName: \"kubernetes.io/projected/3b12758d-18f5-46f4-83b7-c52251978743-kube-api-access-7klw8\") pod \"maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb\" (UID: \"3b12758d-18f5-46f4-83b7-c52251978743\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" Oct 30 12:45:00.449378 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.449262 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aafad9d8-210e-4385-8d3a-447fa25f1d53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f\" (UID: \"aafad9d8-210e-4385-8d3a-447fa25f1d53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" Oct 30 12:45:00.449378 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.449296 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b12758d-18f5-46f4-83b7-c52251978743-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb\" (UID: \"3b12758d-18f5-46f4-83b7-c52251978743\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" Oct 30 12:45:00.456821 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.454257 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" Oct 30 12:45:00.491982 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.491944 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d"] Oct 30 12:45:00.496319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.496283 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g"] Oct 30 12:45:00.502489 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.502460 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" Oct 30 12:45:00.551355 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.551317 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f58a672-ac8b-4715-aa9a-b9d16de76312-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g\" (UID: \"1f58a672-ac8b-4715-aa9a-b9d16de76312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" Oct 30 12:45:00.551551 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.551488 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef6c944-14d1-447f-99bd-550d88053a5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d\" (UID: \"5ef6c944-14d1-447f-99bd-550d88053a5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" Oct 30 12:45:00.551603 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.551557 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-569vj\" (UniqueName: \"kubernetes.io/projected/aafad9d8-210e-4385-8d3a-447fa25f1d53-kube-api-access-569vj\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f\" (UID: \"aafad9d8-210e-4385-8d3a-447fa25f1d53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" Oct 30 12:45:00.551650 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.551626 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7klw8\" (UniqueName: \"kubernetes.io/projected/3b12758d-18f5-46f4-83b7-c52251978743-kube-api-access-7klw8\") pod \"maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb\" (UID: \"3b12758d-18f5-46f4-83b7-c52251978743\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" Oct 30 12:45:00.551690 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.551668 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aafad9d8-210e-4385-8d3a-447fa25f1d53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f\" (UID: \"aafad9d8-210e-4385-8d3a-447fa25f1d53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" Oct 30 12:45:00.551737 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.551719 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b12758d-18f5-46f4-83b7-c52251978743-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb\" (UID: \"3b12758d-18f5-46f4-83b7-c52251978743\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" Oct 30 12:45:00.551810 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.551792 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcnlc\" (UniqueName: \"kubernetes.io/projected/1f58a672-ac8b-4715-aa9a-b9d16de76312-kube-api-access-xcnlc\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g\" (UID: \"1f58a672-ac8b-4715-aa9a-b9d16de76312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" Oct 30 12:45:00.551864 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.551844 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qts6s\" (UniqueName: \"kubernetes.io/projected/5ef6c944-14d1-447f-99bd-550d88053a5c-kube-api-access-qts6s\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d\" (UID: \"5ef6c944-14d1-447f-99bd-550d88053a5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" Oct 30 12:45:00.552445 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.552419 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b12758d-18f5-46f4-83b7-c52251978743-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb\" (UID: \"3b12758d-18f5-46f4-83b7-c52251978743\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" Oct 30 12:45:00.552563 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.552491 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aafad9d8-210e-4385-8d3a-447fa25f1d53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f\" (UID: \"aafad9d8-210e-4385-8d3a-447fa25f1d53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" Oct 30 12:45:00.588443 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.588300 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g"] Oct 30 12:45:00.605739 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.605698 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7klw8\" (UniqueName: \"kubernetes.io/projected/3b12758d-18f5-46f4-83b7-c52251978743-kube-api-access-7klw8\") pod \"maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb\" (UID: \"3b12758d-18f5-46f4-83b7-c52251978743\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" Oct 30 12:45:00.636613 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.636581 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-569vj\" (UniqueName: \"kubernetes.io/projected/aafad9d8-210e-4385-8d3a-447fa25f1d53-kube-api-access-569vj\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f\" (UID: \"aafad9d8-210e-4385-8d3a-447fa25f1d53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" Oct 30 12:45:00.653108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.653052 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef6c944-14d1-447f-99bd-550d88053a5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d\" (UID: \"5ef6c944-14d1-447f-99bd-550d88053a5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" Oct 30 12:45:00.653301 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.653227 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcnlc\" (UniqueName: \"kubernetes.io/projected/1f58a672-ac8b-4715-aa9a-b9d16de76312-kube-api-access-xcnlc\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g\" (UID: \"1f58a672-ac8b-4715-aa9a-b9d16de76312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" Oct 30 12:45:00.653301 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.653253 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qts6s\" (UniqueName: \"kubernetes.io/projected/5ef6c944-14d1-447f-99bd-550d88053a5c-kube-api-access-qts6s\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d\" (UID: \"5ef6c944-14d1-447f-99bd-550d88053a5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" Oct 30 12:45:00.653399 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.653382 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f58a672-ac8b-4715-aa9a-b9d16de76312-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g\" (UID: \"1f58a672-ac8b-4715-aa9a-b9d16de76312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" Oct 30 12:45:00.653871 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.653843 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef6c944-14d1-447f-99bd-550d88053a5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d\" (UID: \"5ef6c944-14d1-447f-99bd-550d88053a5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" Oct 30 12:45:00.654157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.654131 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f58a672-ac8b-4715-aa9a-b9d16de76312-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g\" (UID: \"1f58a672-ac8b-4715-aa9a-b9d16de76312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" Oct 30 12:45:00.654332 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.654316 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" Oct 30 12:45:00.697393 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.697354 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcnlc\" (UniqueName: \"kubernetes.io/projected/1f58a672-ac8b-4715-aa9a-b9d16de76312-kube-api-access-xcnlc\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g\" (UID: \"1f58a672-ac8b-4715-aa9a-b9d16de76312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" Oct 30 12:45:00.697550 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.697424 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qts6s\" (UniqueName: \"kubernetes.io/projected/5ef6c944-14d1-447f-99bd-550d88053a5c-kube-api-access-qts6s\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d\" (UID: \"5ef6c944-14d1-447f-99bd-550d88053a5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" Oct 30 12:45:00.704218 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.704179 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" Oct 30 12:45:00.773641 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.773596 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" Oct 30 12:45:00.818763 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.818719 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" Oct 30 12:45:00.995535 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:00.995473 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb"] Oct 30 12:45:00.999957 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:45:00.999925 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b12758d_18f5_46f4_83b7_c52251978743.slice/crio-718d070f21af0e49ebb5dd59e27e64c5e03b1980dca2f04fe10b8549d8c0dc19 WatchSource:0}: Error finding container 718d070f21af0e49ebb5dd59e27e64c5e03b1980dca2f04fe10b8549d8c0dc19: Status 404 returned error can't find the container with id 718d070f21af0e49ebb5dd59e27e64c5e03b1980dca2f04fe10b8549d8c0dc19 Oct 30 12:45:01.078129 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:01.078092 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f"] Oct 30 12:45:01.078804 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:45:01.078774 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafad9d8_210e_4385_8d3a_447fa25f1d53.slice/crio-ceead1f5381df7db70995c96b9beafa5fbe3d21da894a35c1f06ef7d93944c8d WatchSource:0}: Error finding container ceead1f5381df7db70995c96b9beafa5fbe3d21da894a35c1f06ef7d93944c8d: Status 404 returned error can't find the container with id ceead1f5381df7db70995c96b9beafa5fbe3d21da894a35c1f06ef7d93944c8d Oct 30 12:45:01.128659 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:01.128616 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d"] Oct 30 12:45:01.181462 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:01.181410 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g"] Oct 30 12:45:01.207337 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:45:01.207296 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef6c944_14d1_447f_99bd_550d88053a5c.slice/crio-2f813afd4d6f3c5f0ff906ed8ebd2e6c0c434ab03a147e6b79a44697f726c5d7 WatchSource:0}: Error finding container 2f813afd4d6f3c5f0ff906ed8ebd2e6c0c434ab03a147e6b79a44697f726c5d7: Status 404 returned error can't find the container with id 2f813afd4d6f3c5f0ff906ed8ebd2e6c0c434ab03a147e6b79a44697f726c5d7 Oct 30 12:45:01.207808 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:45:01.207775 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f58a672_ac8b_4715_aa9a_b9d16de76312.slice/crio-93e56ec571d4178ea7cebfa3f0455f363f886d876eb5e8157d640c28be9206cd WatchSource:0}: Error finding container 93e56ec571d4178ea7cebfa3f0455f363f886d876eb5e8157d640c28be9206cd: Status 404 returned error can't find the container with id 93e56ec571d4178ea7cebfa3f0455f363f886d876eb5e8157d640c28be9206cd Oct 30 12:45:01.943238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:01.943197 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" event={"ID":"1f58a672-ac8b-4715-aa9a-b9d16de76312","Type":"ContainerStarted","Data":"05b4cdac144893fd959cd8c92ddad10f708558fcfd06a7eda24b5ebe9078cc39"} Oct 30 12:45:01.943699 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:01.943258 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" event={"ID":"1f58a672-ac8b-4715-aa9a-b9d16de76312","Type":"ContainerStarted","Data":"93e56ec571d4178ea7cebfa3f0455f363f886d876eb5e8157d640c28be9206cd"} Oct 30 12:45:01.945065 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:01.945040 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" event={"ID":"5ef6c944-14d1-447f-99bd-550d88053a5c","Type":"ContainerStarted","Data":"fe60d6f2be10b877a81414e25a9d43331c08d6f59f1ab51d93ad5e368da50ee7"} Oct 30 12:45:01.945207 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:01.945104 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" event={"ID":"5ef6c944-14d1-447f-99bd-550d88053a5c","Type":"ContainerStarted","Data":"2f813afd4d6f3c5f0ff906ed8ebd2e6c0c434ab03a147e6b79a44697f726c5d7"} Oct 30 12:45:01.946735 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:01.946711 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" event={"ID":"aafad9d8-210e-4385-8d3a-447fa25f1d53","Type":"ContainerStarted","Data":"6ef7f50c7651e10fd73a34012a3998ecba6234994b615d980236f97c32422d6b"} Oct 30 12:45:01.946829 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:01.946754 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" event={"ID":"aafad9d8-210e-4385-8d3a-447fa25f1d53","Type":"ContainerStarted","Data":"ceead1f5381df7db70995c96b9beafa5fbe3d21da894a35c1f06ef7d93944c8d"} Oct 30 12:45:01.960026 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:01.959982 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" event={"ID":"3b12758d-18f5-46f4-83b7-c52251978743","Type":"ContainerStarted","Data":"2cedd989f49102ec7612de5eed559faf3429f420b0a45fe53f6ee1f7e32a0569"} Oct 30 12:45:01.960267 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:01.960117 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" event={"ID":"3b12758d-18f5-46f4-83b7-c52251978743","Type":"ContainerStarted","Data":"718d070f21af0e49ebb5dd59e27e64c5e03b1980dca2f04fe10b8549d8c0dc19"} Oct 30 12:45:01.992818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:01.992756 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" podStartSLOduration=1.992737166 podStartE2EDuration="1.992737166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:01.992265446 +0000 UTC m=+3053.026403842" watchObservedRunningTime="2026-10-30 12:45:01.992737166 +0000 UTC m=+3053.026875564" Oct 30 12:45:02.042373 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:02.042316 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" podStartSLOduration=2.042298983 podStartE2EDuration="2.042298983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:02.039203592 +0000 UTC m=+3053.073341988" watchObservedRunningTime="2026-10-30 12:45:02.042298983 +0000 UTC m=+3053.076437379" Oct 30 12:45:02.091450 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:02.091330 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" podStartSLOduration=2.091315206 podStartE2EDuration="2.091315206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:02.090249509 +0000 UTC m=+3053.124387904" watchObservedRunningTime="2026-10-30 12:45:02.091315206 +0000 UTC m=+3053.125453627" Oct 30 12:45:02.140145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:02.140065 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" podStartSLOduration=2.140049437 podStartE2EDuration="2.140049437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:02.137736232 +0000 UTC m=+3053.171874628" watchObservedRunningTime="2026-10-30 12:45:02.140049437 +0000 UTC m=+3053.174187832" Oct 30 12:45:02.970743 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:02.970706 4468 generic.go:356] "Generic (PLEG): container finished" podID="aafad9d8-210e-4385-8d3a-447fa25f1d53" containerID="6ef7f50c7651e10fd73a34012a3998ecba6234994b615d980236f97c32422d6b" exitCode=60 Oct 30 12:45:02.971280 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:02.970803 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" event={"ID":"aafad9d8-210e-4385-8d3a-447fa25f1d53","Type":"ContainerDied","Data":"6ef7f50c7651e10fd73a34012a3998ecba6234994b615d980236f97c32422d6b"} Oct 30 12:45:02.971280 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:02.971203 4468 scope.go:122] "RemoveContainer" containerID="6ef7f50c7651e10fd73a34012a3998ecba6234994b615d980236f97c32422d6b" Oct 30 12:45:02.973038 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:02.973002 4468 generic.go:356] "Generic (PLEG): container finished" podID="3b12758d-18f5-46f4-83b7-c52251978743" containerID="2cedd989f49102ec7612de5eed559faf3429f420b0a45fe53f6ee1f7e32a0569" exitCode=60 Oct 30 12:45:02.973269 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:02.973238 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" event={"ID":"3b12758d-18f5-46f4-83b7-c52251978743","Type":"ContainerDied","Data":"2cedd989f49102ec7612de5eed559faf3429f420b0a45fe53f6ee1f7e32a0569"} Oct 30 12:45:02.973672 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:02.973564 4468 scope.go:122] "RemoveContainer" containerID="2cedd989f49102ec7612de5eed559faf3429f420b0a45fe53f6ee1f7e32a0569" Oct 30 12:45:02.977458 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:02.975569 4468 generic.go:356] "Generic (PLEG): container finished" podID="1f58a672-ac8b-4715-aa9a-b9d16de76312" containerID="05b4cdac144893fd959cd8c92ddad10f708558fcfd06a7eda24b5ebe9078cc39" exitCode=60 Oct 30 12:45:02.977458 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:02.975655 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" event={"ID":"1f58a672-ac8b-4715-aa9a-b9d16de76312","Type":"ContainerDied","Data":"05b4cdac144893fd959cd8c92ddad10f708558fcfd06a7eda24b5ebe9078cc39"} Oct 30 12:45:02.977458 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:02.975978 4468 scope.go:122] "RemoveContainer" containerID="05b4cdac144893fd959cd8c92ddad10f708558fcfd06a7eda24b5ebe9078cc39" Oct 30 12:45:02.987857 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:02.987822 4468 generic.go:356] "Generic (PLEG): container finished" podID="5ef6c944-14d1-447f-99bd-550d88053a5c" containerID="fe60d6f2be10b877a81414e25a9d43331c08d6f59f1ab51d93ad5e368da50ee7" exitCode=60 Oct 30 12:45:02.988003 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:02.987874 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" event={"ID":"5ef6c944-14d1-447f-99bd-550d88053a5c","Type":"ContainerDied","Data":"fe60d6f2be10b877a81414e25a9d43331c08d6f59f1ab51d93ad5e368da50ee7"} Oct 30 12:45:02.988312 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:02.988290 4468 scope.go:122] "RemoveContainer" containerID="fe60d6f2be10b877a81414e25a9d43331c08d6f59f1ab51d93ad5e368da50ee7" Oct 30 12:45:03.998380 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:03.998347 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" event={"ID":"1f58a672-ac8b-4715-aa9a-b9d16de76312","Type":"ContainerStarted","Data":"1246e5f2f97ee870e0936c4a5a502d3dd1dbd48c6fb2d650e08c87ac9aa6735a"} Oct 30 12:45:04.000365 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:04.000337 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" event={"ID":"5ef6c944-14d1-447f-99bd-550d88053a5c","Type":"ContainerStarted","Data":"979f28a2d2cafb2126f928527b0d36574b5576b0189f90690e02538c0ac74a1c"} Oct 30 12:45:04.002400 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:04.002372 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" event={"ID":"aafad9d8-210e-4385-8d3a-447fa25f1d53","Type":"ContainerStarted","Data":"2bb8b7e73f4a2df40616584236a2d051b789519cab2d5a3f5ef8db0d13d7ce71"} Oct 30 12:45:04.006237 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:04.006205 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" event={"ID":"3b12758d-18f5-46f4-83b7-c52251978743","Type":"ContainerStarted","Data":"2112f95a6a7520e509f72be71ac7253bccb1119a03ccb2acffb24bb32b1ee64c"} Oct 30 12:45:05.017931 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:05.017895 4468 generic.go:356] "Generic (PLEG): container finished" podID="3b12758d-18f5-46f4-83b7-c52251978743" containerID="2112f95a6a7520e509f72be71ac7253bccb1119a03ccb2acffb24bb32b1ee64c" exitCode=60 Oct 30 12:45:05.018425 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:05.017989 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" event={"ID":"3b12758d-18f5-46f4-83b7-c52251978743","Type":"ContainerDied","Data":"2112f95a6a7520e509f72be71ac7253bccb1119a03ccb2acffb24bb32b1ee64c"} Oct 30 12:45:05.018425 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:05.018056 4468 scope.go:122] "RemoveContainer" containerID="2cedd989f49102ec7612de5eed559faf3429f420b0a45fe53f6ee1f7e32a0569" Oct 30 12:45:05.018425 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:05.018319 4468 scope.go:122] "RemoveContainer" containerID="2112f95a6a7520e509f72be71ac7253bccb1119a03ccb2acffb24bb32b1ee64c" Oct 30 12:45:05.018688 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:45:05.018670 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb_odh-ai-gateway-infra(3b12758d-18f5-46f4-83b7-c52251978743)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" podUID="3b12758d-18f5-46f4-83b7-c52251978743" Oct 30 12:45:05.020778 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:05.020751 4468 generic.go:356] "Generic (PLEG): container finished" podID="1f58a672-ac8b-4715-aa9a-b9d16de76312" containerID="1246e5f2f97ee870e0936c4a5a502d3dd1dbd48c6fb2d650e08c87ac9aa6735a" exitCode=60 Oct 30 12:45:05.020884 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:05.020824 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" event={"ID":"1f58a672-ac8b-4715-aa9a-b9d16de76312","Type":"ContainerDied","Data":"1246e5f2f97ee870e0936c4a5a502d3dd1dbd48c6fb2d650e08c87ac9aa6735a"} Oct 30 12:45:05.021166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:05.021151 4468 scope.go:122] "RemoveContainer" containerID="1246e5f2f97ee870e0936c4a5a502d3dd1dbd48c6fb2d650e08c87ac9aa6735a" Oct 30 12:45:05.021447 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:45:05.021428 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g_odh-ai-gateway-infra(1f58a672-ac8b-4715-aa9a-b9d16de76312)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" podUID="1f58a672-ac8b-4715-aa9a-b9d16de76312" Oct 30 12:45:05.022837 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:05.022815 4468 generic.go:356] "Generic (PLEG): container finished" podID="5ef6c944-14d1-447f-99bd-550d88053a5c" containerID="979f28a2d2cafb2126f928527b0d36574b5576b0189f90690e02538c0ac74a1c" exitCode=60 Oct 30 12:45:05.022957 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:05.022890 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" event={"ID":"5ef6c944-14d1-447f-99bd-550d88053a5c","Type":"ContainerDied","Data":"979f28a2d2cafb2126f928527b0d36574b5576b0189f90690e02538c0ac74a1c"} Oct 30 12:45:05.023143 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:05.023129 4468 scope.go:122] "RemoveContainer" containerID="979f28a2d2cafb2126f928527b0d36574b5576b0189f90690e02538c0ac74a1c" Oct 30 12:45:05.023424 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:45:05.023387 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d_odh-ai-gateway-infra(5ef6c944-14d1-447f-99bd-550d88053a5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" podUID="5ef6c944-14d1-447f-99bd-550d88053a5c" Oct 30 12:45:05.024918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:05.024896 4468 generic.go:356] "Generic (PLEG): container finished" podID="aafad9d8-210e-4385-8d3a-447fa25f1d53" containerID="2bb8b7e73f4a2df40616584236a2d051b789519cab2d5a3f5ef8db0d13d7ce71" exitCode=60 Oct 30 12:45:05.025006 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:05.024966 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" event={"ID":"aafad9d8-210e-4385-8d3a-447fa25f1d53","Type":"ContainerDied","Data":"2bb8b7e73f4a2df40616584236a2d051b789519cab2d5a3f5ef8db0d13d7ce71"} Oct 30 12:45:05.025328 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:05.025310 4468 scope.go:122] "RemoveContainer" containerID="2bb8b7e73f4a2df40616584236a2d051b789519cab2d5a3f5ef8db0d13d7ce71" Oct 30 12:45:05.025644 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:45:05.025623 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f_odh-ai-gateway-infra(aafad9d8-210e-4385-8d3a-447fa25f1d53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" podUID="aafad9d8-210e-4385-8d3a-447fa25f1d53" Oct 30 12:45:05.032553 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:05.032532 4468 scope.go:122] "RemoveContainer" containerID="05b4cdac144893fd959cd8c92ddad10f708558fcfd06a7eda24b5ebe9078cc39" Oct 30 12:45:05.046876 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:05.046852 4468 scope.go:122] "RemoveContainer" containerID="fe60d6f2be10b877a81414e25a9d43331c08d6f59f1ab51d93ad5e368da50ee7" Oct 30 12:45:05.061680 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:05.061657 4468 scope.go:122] "RemoveContainer" containerID="6ef7f50c7651e10fd73a34012a3998ecba6234994b615d980236f97c32422d6b" Oct 30 12:45:06.035596 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:06.035562 4468 scope.go:122] "RemoveContainer" containerID="979f28a2d2cafb2126f928527b0d36574b5576b0189f90690e02538c0ac74a1c" Oct 30 12:45:06.036139 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:45:06.035894 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d_odh-ai-gateway-infra(5ef6c944-14d1-447f-99bd-550d88053a5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" podUID="5ef6c944-14d1-447f-99bd-550d88053a5c" Oct 30 12:45:06.037526 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:06.037501 4468 scope.go:122] "RemoveContainer" containerID="2bb8b7e73f4a2df40616584236a2d051b789519cab2d5a3f5ef8db0d13d7ce71" Oct 30 12:45:06.037839 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:45:06.037816 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f_odh-ai-gateway-infra(aafad9d8-210e-4385-8d3a-447fa25f1d53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" podUID="aafad9d8-210e-4385-8d3a-447fa25f1d53" Oct 30 12:45:06.039339 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:06.039317 4468 scope.go:122] "RemoveContainer" containerID="2112f95a6a7520e509f72be71ac7253bccb1119a03ccb2acffb24bb32b1ee64c" Oct 30 12:45:06.039638 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:45:06.039604 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb_odh-ai-gateway-infra(3b12758d-18f5-46f4-83b7-c52251978743)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" podUID="3b12758d-18f5-46f4-83b7-c52251978743" Oct 30 12:45:06.041100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:06.041082 4468 scope.go:122] "RemoveContainer" containerID="1246e5f2f97ee870e0936c4a5a502d3dd1dbd48c6fb2d650e08c87ac9aa6735a" Oct 30 12:45:06.041306 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:45:06.041288 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g_odh-ai-gateway-infra(1f58a672-ac8b-4715-aa9a-b9d16de76312)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" podUID="1f58a672-ac8b-4715-aa9a-b9d16de76312" Oct 30 12:45:15.082410 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:15.082377 4468 scope.go:122] "RemoveContainer" containerID="979f28a2d2cafb2126f928527b0d36574b5576b0189f90690e02538c0ac74a1c" Oct 30 12:45:16.083305 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:16.083271 4468 scope.go:122] "RemoveContainer" containerID="2112f95a6a7520e509f72be71ac7253bccb1119a03ccb2acffb24bb32b1ee64c" Oct 30 12:45:16.083749 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:16.083337 4468 scope.go:122] "RemoveContainer" containerID="1246e5f2f97ee870e0936c4a5a502d3dd1dbd48c6fb2d650e08c87ac9aa6735a" Oct 30 12:45:16.121478 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:16.121444 4468 generic.go:356] "Generic (PLEG): container finished" podID="5ef6c944-14d1-447f-99bd-550d88053a5c" containerID="6c797d68782896fbd6e8fc929a166408c72dcb1e23c5ee50693351f7c742257f" exitCode=60 Oct 30 12:45:16.121654 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:16.121509 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" event={"ID":"5ef6c944-14d1-447f-99bd-550d88053a5c","Type":"ContainerDied","Data":"6c797d68782896fbd6e8fc929a166408c72dcb1e23c5ee50693351f7c742257f"} Oct 30 12:45:16.121654 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:16.121569 4468 scope.go:122] "RemoveContainer" containerID="979f28a2d2cafb2126f928527b0d36574b5576b0189f90690e02538c0ac74a1c" Oct 30 12:45:16.122033 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:16.122013 4468 scope.go:122] "RemoveContainer" containerID="6c797d68782896fbd6e8fc929a166408c72dcb1e23c5ee50693351f7c742257f" Oct 30 12:45:16.122434 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:45:16.122413 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d_odh-ai-gateway-infra(5ef6c944-14d1-447f-99bd-550d88053a5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" podUID="5ef6c944-14d1-447f-99bd-550d88053a5c" Oct 30 12:45:17.133886 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:17.133826 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" event={"ID":"3b12758d-18f5-46f4-83b7-c52251978743","Type":"ContainerStarted","Data":"d8303e3cdb144258f5f5e5e35c5cf24e5273670e3ae4c23bc9ff99d780db790c"} Oct 30 12:45:17.135994 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:17.135959 4468 generic.go:356] "Generic (PLEG): container finished" podID="1f58a672-ac8b-4715-aa9a-b9d16de76312" containerID="8874661e566586535c47c879ebea6c7328727a97143819bfea1ade56f8d8ede0" exitCode=60 Oct 30 12:45:17.136169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:17.136025 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" event={"ID":"1f58a672-ac8b-4715-aa9a-b9d16de76312","Type":"ContainerDied","Data":"8874661e566586535c47c879ebea6c7328727a97143819bfea1ade56f8d8ede0"} Oct 30 12:45:17.136169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:17.136099 4468 scope.go:122] "RemoveContainer" containerID="1246e5f2f97ee870e0936c4a5a502d3dd1dbd48c6fb2d650e08c87ac9aa6735a" Oct 30 12:45:17.136507 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:17.136490 4468 scope.go:122] "RemoveContainer" containerID="8874661e566586535c47c879ebea6c7328727a97143819bfea1ade56f8d8ede0" Oct 30 12:45:17.136900 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:45:17.136865 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g_odh-ai-gateway-infra(1f58a672-ac8b-4715-aa9a-b9d16de76312)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" podUID="1f58a672-ac8b-4715-aa9a-b9d16de76312" Oct 30 12:45:17.192248 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:17.192209 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d"] Oct 30 12:45:17.374322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:17.374249 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" Oct 30 12:45:17.414858 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:17.414824 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef6c944-14d1-447f-99bd-550d88053a5c-kube-api-access-qts6s\" (UniqueName: \"kubernetes.io/projected/5ef6c944-14d1-447f-99bd-550d88053a5c-kube-api-access-qts6s\") pod \"5ef6c944-14d1-447f-99bd-550d88053a5c\" (UID: \"5ef6c944-14d1-447f-99bd-550d88053a5c\") " Oct 30 12:45:17.415114 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:17.414948 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ef6c944-14d1-447f-99bd-550d88053a5c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef6c944-14d1-447f-99bd-550d88053a5c-maas-api-service-ca\") pod \"5ef6c944-14d1-447f-99bd-550d88053a5c\" (UID: \"5ef6c944-14d1-447f-99bd-550d88053a5c\") " Oct 30 12:45:17.415344 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:17.415316 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef6c944-14d1-447f-99bd-550d88053a5c-maas-api-service-ca" pod "5ef6c944-14d1-447f-99bd-550d88053a5c" (UID: "5ef6c944-14d1-447f-99bd-550d88053a5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:17.415678 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:17.415656 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef6c944-14d1-447f-99bd-550d88053a5c-maas-api-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:17.417169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:17.417144 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef6c944-14d1-447f-99bd-550d88053a5c-kube-api-access-qts6s" pod "5ef6c944-14d1-447f-99bd-550d88053a5c" (UID: "5ef6c944-14d1-447f-99bd-550d88053a5c"). InnerVolumeSpecName "kube-api-access-qts6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:17.516748 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:17.516703 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qts6s\" (UniqueName: \"kubernetes.io/projected/5ef6c944-14d1-447f-99bd-550d88053a5c-kube-api-access-qts6s\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:18.082424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:18.082388 4468 scope.go:122] "RemoveContainer" containerID="2bb8b7e73f4a2df40616584236a2d051b789519cab2d5a3f5ef8db0d13d7ce71" Oct 30 12:45:18.145479 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:18.145446 4468 generic.go:356] "Generic (PLEG): container finished" podID="3b12758d-18f5-46f4-83b7-c52251978743" containerID="d8303e3cdb144258f5f5e5e35c5cf24e5273670e3ae4c23bc9ff99d780db790c" exitCode=60 Oct 30 12:45:18.145912 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:18.145519 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" event={"ID":"3b12758d-18f5-46f4-83b7-c52251978743","Type":"ContainerDied","Data":"d8303e3cdb144258f5f5e5e35c5cf24e5273670e3ae4c23bc9ff99d780db790c"} Oct 30 12:45:18.145912 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:18.145595 4468 scope.go:122] "RemoveContainer" containerID="2112f95a6a7520e509f72be71ac7253bccb1119a03ccb2acffb24bb32b1ee64c" Oct 30 12:45:18.146062 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:18.146035 4468 scope.go:122] "RemoveContainer" containerID="d8303e3cdb144258f5f5e5e35c5cf24e5273670e3ae4c23bc9ff99d780db790c" Oct 30 12:45:18.146504 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:45:18.146466 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb_odh-ai-gateway-infra(3b12758d-18f5-46f4-83b7-c52251978743)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" podUID="3b12758d-18f5-46f4-83b7-c52251978743" Oct 30 12:45:18.148857 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:18.148839 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" Oct 30 12:45:18.148976 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:18.148877 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d" event={"ID":"5ef6c944-14d1-447f-99bd-550d88053a5c","Type":"ContainerDied","Data":"2f813afd4d6f3c5f0ff906ed8ebd2e6c0c434ab03a147e6b79a44697f726c5d7"} Oct 30 12:45:18.163910 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:18.163888 4468 scope.go:122] "RemoveContainer" containerID="6c797d68782896fbd6e8fc929a166408c72dcb1e23c5ee50693351f7c742257f" Oct 30 12:45:18.238506 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:18.238456 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb"] Oct 30 12:45:18.256182 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:18.256141 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d"] Oct 30 12:45:18.264258 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:18.264213 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889405-7tg8d"] Oct 30 12:45:18.275199 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:18.275164 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g"] Oct 30 12:45:18.511515 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:18.511486 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" Oct 30 12:45:18.531978 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:18.531942 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f58a672-ac8b-4715-aa9a-b9d16de76312-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f58a672-ac8b-4715-aa9a-b9d16de76312-maas-api-service-ca\") pod \"1f58a672-ac8b-4715-aa9a-b9d16de76312\" (UID: \"1f58a672-ac8b-4715-aa9a-b9d16de76312\") " Oct 30 12:45:18.532134 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:18.531996 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f58a672-ac8b-4715-aa9a-b9d16de76312-kube-api-access-xcnlc\" (UniqueName: \"kubernetes.io/projected/1f58a672-ac8b-4715-aa9a-b9d16de76312-kube-api-access-xcnlc\") pod \"1f58a672-ac8b-4715-aa9a-b9d16de76312\" (UID: \"1f58a672-ac8b-4715-aa9a-b9d16de76312\") " Oct 30 12:45:18.532345 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:18.532323 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f58a672-ac8b-4715-aa9a-b9d16de76312-maas-api-service-ca" pod "1f58a672-ac8b-4715-aa9a-b9d16de76312" (UID: "1f58a672-ac8b-4715-aa9a-b9d16de76312"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:18.532524 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:18.532506 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f58a672-ac8b-4715-aa9a-b9d16de76312-maas-api-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:18.534112 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:18.534054 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f58a672-ac8b-4715-aa9a-b9d16de76312-kube-api-access-xcnlc" pod "1f58a672-ac8b-4715-aa9a-b9d16de76312" (UID: "1f58a672-ac8b-4715-aa9a-b9d16de76312"). InnerVolumeSpecName "kube-api-access-xcnlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:18.633869 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:18.633783 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcnlc\" (UniqueName: \"kubernetes.io/projected/1f58a672-ac8b-4715-aa9a-b9d16de76312-kube-api-access-xcnlc\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:19.092405 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:19.092361 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef6c944-14d1-447f-99bd-550d88053a5c" path="/var/lib/kubelet/pods/5ef6c944-14d1-447f-99bd-550d88053a5c/volumes" Oct 30 12:45:19.160234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:19.160196 4468 generic.go:356] "Generic (PLEG): container finished" podID="aafad9d8-210e-4385-8d3a-447fa25f1d53" containerID="160306f4ba66b8e331f4480a6c7b512440f66bf728a8df479b4e4fb009285296" exitCode=60 Oct 30 12:45:19.160963 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:19.160934 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" event={"ID":"aafad9d8-210e-4385-8d3a-447fa25f1d53","Type":"ContainerDied","Data":"160306f4ba66b8e331f4480a6c7b512440f66bf728a8df479b4e4fb009285296"} Oct 30 12:45:19.161171 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:19.161156 4468 scope.go:122] "RemoveContainer" containerID="2bb8b7e73f4a2df40616584236a2d051b789519cab2d5a3f5ef8db0d13d7ce71" Oct 30 12:45:19.161810 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:19.161789 4468 scope.go:122] "RemoveContainer" containerID="160306f4ba66b8e331f4480a6c7b512440f66bf728a8df479b4e4fb009285296" Oct 30 12:45:19.162735 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:45:19.162707 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f_odh-ai-gateway-infra(aafad9d8-210e-4385-8d3a-447fa25f1d53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" podUID="aafad9d8-210e-4385-8d3a-447fa25f1d53" Oct 30 12:45:19.194447 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:19.191405 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" event={"ID":"1f58a672-ac8b-4715-aa9a-b9d16de76312","Type":"ContainerDied","Data":"93e56ec571d4178ea7cebfa3f0455f363f886d876eb5e8157d640c28be9206cd"} Oct 30 12:45:19.194447 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:19.193661 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g" Oct 30 12:45:19.202949 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:19.202919 4468 scope.go:122] "RemoveContainer" containerID="8874661e566586535c47c879ebea6c7328727a97143819bfea1ade56f8d8ede0" Oct 30 12:45:19.337166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:19.337065 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g"] Oct 30 12:45:19.342400 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:19.342369 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889405-4tn4g"] Oct 30 12:45:19.370116 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:19.370042 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" Oct 30 12:45:19.446258 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:19.446222 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b12758d-18f5-46f4-83b7-c52251978743-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b12758d-18f5-46f4-83b7-c52251978743-maas-api-service-ca\") pod \"3b12758d-18f5-46f4-83b7-c52251978743\" (UID: \"3b12758d-18f5-46f4-83b7-c52251978743\") " Oct 30 12:45:19.446438 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:19.446422 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b12758d-18f5-46f4-83b7-c52251978743-kube-api-access-7klw8\" (UniqueName: \"kubernetes.io/projected/3b12758d-18f5-46f4-83b7-c52251978743-kube-api-access-7klw8\") pod \"3b12758d-18f5-46f4-83b7-c52251978743\" (UID: \"3b12758d-18f5-46f4-83b7-c52251978743\") " Oct 30 12:45:19.446674 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:19.446646 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b12758d-18f5-46f4-83b7-c52251978743-maas-api-service-ca" pod "3b12758d-18f5-46f4-83b7-c52251978743" (UID: "3b12758d-18f5-46f4-83b7-c52251978743"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:19.447087 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:19.447049 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b12758d-18f5-46f4-83b7-c52251978743-maas-api-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:19.449486 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:19.449454 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b12758d-18f5-46f4-83b7-c52251978743-kube-api-access-7klw8" pod "3b12758d-18f5-46f4-83b7-c52251978743" (UID: "3b12758d-18f5-46f4-83b7-c52251978743"). InnerVolumeSpecName "kube-api-access-7klw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:19.548416 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:19.548324 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7klw8\" (UniqueName: \"kubernetes.io/projected/3b12758d-18f5-46f4-83b7-c52251978743-kube-api-access-7klw8\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:20.207401 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:20.207364 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" event={"ID":"3b12758d-18f5-46f4-83b7-c52251978743","Type":"ContainerDied","Data":"718d070f21af0e49ebb5dd59e27e64c5e03b1980dca2f04fe10b8549d8c0dc19"} Oct 30 12:45:20.207401 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:20.207393 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb" Oct 30 12:45:20.207916 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:20.207433 4468 scope.go:122] "RemoveContainer" containerID="d8303e3cdb144258f5f5e5e35c5cf24e5273670e3ae4c23bc9ff99d780db790c" Oct 30 12:45:20.256048 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:20.256005 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f"] Oct 30 12:45:20.288505 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:20.288467 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb"] Oct 30 12:45:20.301905 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:20.301868 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a786af-29889405-dg2zb"] Oct 30 12:45:20.437294 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:20.437264 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" Oct 30 12:45:20.462733 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:20.462637 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aafad9d8-210e-4385-8d3a-447fa25f1d53-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aafad9d8-210e-4385-8d3a-447fa25f1d53-maas-api-service-ca\") pod \"aafad9d8-210e-4385-8d3a-447fa25f1d53\" (UID: \"aafad9d8-210e-4385-8d3a-447fa25f1d53\") " Oct 30 12:45:20.462896 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:20.462869 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aafad9d8-210e-4385-8d3a-447fa25f1d53-kube-api-access-569vj\" (UniqueName: \"kubernetes.io/projected/aafad9d8-210e-4385-8d3a-447fa25f1d53-kube-api-access-569vj\") pod \"aafad9d8-210e-4385-8d3a-447fa25f1d53\" (UID: \"aafad9d8-210e-4385-8d3a-447fa25f1d53\") " Oct 30 12:45:20.463137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:20.463108 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aafad9d8-210e-4385-8d3a-447fa25f1d53-maas-api-service-ca" pod "aafad9d8-210e-4385-8d3a-447fa25f1d53" (UID: "aafad9d8-210e-4385-8d3a-447fa25f1d53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:20.463417 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:20.463402 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aafad9d8-210e-4385-8d3a-447fa25f1d53-maas-api-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:20.465037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:20.465011 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafad9d8-210e-4385-8d3a-447fa25f1d53-kube-api-access-569vj" pod "aafad9d8-210e-4385-8d3a-447fa25f1d53" (UID: "aafad9d8-210e-4385-8d3a-447fa25f1d53"). InnerVolumeSpecName "kube-api-access-569vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:20.564793 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:20.564751 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-569vj\" (UniqueName: \"kubernetes.io/projected/aafad9d8-210e-4385-8d3a-447fa25f1d53-kube-api-access-569vj\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:21.091918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:21.091880 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f58a672-ac8b-4715-aa9a-b9d16de76312" path="/var/lib/kubelet/pods/1f58a672-ac8b-4715-aa9a-b9d16de76312/volumes" Oct 30 12:45:21.092519 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:21.092487 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b12758d-18f5-46f4-83b7-c52251978743" path="/var/lib/kubelet/pods/3b12758d-18f5-46f4-83b7-c52251978743/volumes" Oct 30 12:45:21.219836 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:21.219791 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" event={"ID":"aafad9d8-210e-4385-8d3a-447fa25f1d53","Type":"ContainerDied","Data":"ceead1f5381df7db70995c96b9beafa5fbe3d21da894a35c1f06ef7d93944c8d"} Oct 30 12:45:21.220387 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:21.219887 4468 scope.go:122] "RemoveContainer" containerID="160306f4ba66b8e331f4480a6c7b512440f66bf728a8df479b4e4fb009285296" Oct 30 12:45:21.220488 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:21.219819 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f" Oct 30 12:45:21.310434 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:21.310358 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f"] Oct 30 12:45:21.376184 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:21.376092 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dbe123-29889405-pzd2f"] Oct 30 12:45:23.092057 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:23.092016 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aafad9d8-210e-4385-8d3a-447fa25f1d53" path="/var/lib/kubelet/pods/aafad9d8-210e-4385-8d3a-447fa25f1d53/volumes" Oct 30 12:45:23.600775 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:45:23.600740 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:45:27.121317 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:27.119838 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd"] Oct 30 12:45:27.121317 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:27.120140 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" podUID="73e2eab8-2195-41e0-97a7-63383ccbbe48" containerName="istio-proxy" containerID="cri-o://c5c50ce3d3ce81f5687ab0de35881a9ca9075703802cc431f0f0bf62127e30c4" gracePeriod=30 Oct 30 12:45:29.741118 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:29.741059 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt"] Oct 30 12:45:29.741605 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:29.741357 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" podUID="b7abde5f-37e2-40d0-9ddc-bd907e115aa0" containerName="istio-proxy" containerID="cri-o://a50e1660fc0db064e39bfea6e4a5c6743762d77870e738cf16ad987d9cf4f198" gracePeriod=30 Oct 30 12:45:32.320334 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.320300 4468 generic.go:356] "Generic (PLEG): container finished" podID="73e2eab8-2195-41e0-97a7-63383ccbbe48" containerID="c5c50ce3d3ce81f5687ab0de35881a9ca9075703802cc431f0f0bf62127e30c4" exitCode=0 Oct 30 12:45:32.320809 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.320368 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" event={"ID":"73e2eab8-2195-41e0-97a7-63383ccbbe48","Type":"ContainerDied","Data":"c5c50ce3d3ce81f5687ab0de35881a9ca9075703802cc431f0f0bf62127e30c4"} Oct 30 12:45:32.320809 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.320443 4468 scope.go:122] "RemoveContainer" containerID="454ccb1c17bbc3e0b94fc856c34c9b6f5ce6efcd14552010891686d4a0fc55b2" Oct 30 12:45:32.339222 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.339193 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv"] Oct 30 12:45:32.340464 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340417 4468 cpu_manager.go:414] "containerMap: removing container" podUID="aafad9d8-210e-4385-8d3a-447fa25f1d53" containerName="cleanup" Oct 30 12:45:32.340464 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340462 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafad9d8-210e-4385-8d3a-447fa25f1d53" containerName="cleanup" Oct 30 12:45:32.340631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340488 4468 cpu_manager.go:414] "containerMap: removing container" podUID="5ef6c944-14d1-447f-99bd-550d88053a5c" containerName="cleanup" Oct 30 12:45:32.340631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340496 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef6c944-14d1-447f-99bd-550d88053a5c" containerName="cleanup" Oct 30 12:45:32.340631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340508 4468 cpu_manager.go:414] "containerMap: removing container" podUID="1f58a672-ac8b-4715-aa9a-b9d16de76312" containerName="cleanup" Oct 30 12:45:32.340631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340516 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f58a672-ac8b-4715-aa9a-b9d16de76312" containerName="cleanup" Oct 30 12:45:32.340631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340528 4468 cpu_manager.go:414] "containerMap: removing container" podUID="1f58a672-ac8b-4715-aa9a-b9d16de76312" containerName="cleanup" Oct 30 12:45:32.340631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340535 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f58a672-ac8b-4715-aa9a-b9d16de76312" containerName="cleanup" Oct 30 12:45:32.340631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340566 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3b12758d-18f5-46f4-83b7-c52251978743" containerName="cleanup" Oct 30 12:45:32.340631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340573 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b12758d-18f5-46f4-83b7-c52251978743" containerName="cleanup" Oct 30 12:45:32.340631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340589 4468 cpu_manager.go:414] "containerMap: removing container" podUID="5ef6c944-14d1-447f-99bd-550d88053a5c" containerName="cleanup" Oct 30 12:45:32.340631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340596 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef6c944-14d1-447f-99bd-550d88053a5c" containerName="cleanup" Oct 30 12:45:32.340631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340611 4468 cpu_manager.go:414] "containerMap: removing container" podUID="aafad9d8-210e-4385-8d3a-447fa25f1d53" containerName="cleanup" Oct 30 12:45:32.340631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340618 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafad9d8-210e-4385-8d3a-447fa25f1d53" containerName="cleanup" Oct 30 12:45:32.341243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340638 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3b12758d-18f5-46f4-83b7-c52251978743" containerName="cleanup" Oct 30 12:45:32.341243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340645 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b12758d-18f5-46f4-83b7-c52251978743" containerName="cleanup" Oct 30 12:45:32.341243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340656 4468 cpu_manager.go:414] "containerMap: removing container" podUID="5ef6c944-14d1-447f-99bd-550d88053a5c" containerName="cleanup" Oct 30 12:45:32.341243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340663 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef6c944-14d1-447f-99bd-550d88053a5c" containerName="cleanup" Oct 30 12:45:32.341243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340702 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3b12758d-18f5-46f4-83b7-c52251978743" containerName="cleanup" Oct 30 12:45:32.341243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340709 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b12758d-18f5-46f4-83b7-c52251978743" containerName="cleanup" Oct 30 12:45:32.341243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340887 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="aafad9d8-210e-4385-8d3a-447fa25f1d53" containerName="cleanup" Oct 30 12:45:32.341243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340903 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef6c944-14d1-447f-99bd-550d88053a5c" containerName="cleanup" Oct 30 12:45:32.341243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340914 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f58a672-ac8b-4715-aa9a-b9d16de76312" containerName="cleanup" Oct 30 12:45:32.341243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340922 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f58a672-ac8b-4715-aa9a-b9d16de76312" containerName="cleanup" Oct 30 12:45:32.341243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340938 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b12758d-18f5-46f4-83b7-c52251978743" containerName="cleanup" Oct 30 12:45:32.341243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340946 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef6c944-14d1-447f-99bd-550d88053a5c" containerName="cleanup" Oct 30 12:45:32.341243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340983 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="aafad9d8-210e-4385-8d3a-447fa25f1d53" containerName="cleanup" Oct 30 12:45:32.341243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.340994 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef6c944-14d1-447f-99bd-550d88053a5c" containerName="cleanup" Oct 30 12:45:32.341243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.341009 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b12758d-18f5-46f4-83b7-c52251978743" containerName="cleanup" Oct 30 12:45:32.346807 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.346785 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" Oct 30 12:45:32.369478 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.369381 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmlh6\" (UniqueName: \"kubernetes.io/projected/da588bf8-ee9a-4103-8da6-45bc7752c3bb-kube-api-access-hmlh6\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv\" (UID: \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" Oct 30 12:45:32.369643 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.369559 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da588bf8-ee9a-4103-8da6-45bc7752c3bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv\" (UID: \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" Oct 30 12:45:32.369710 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.369659 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da588bf8-ee9a-4103-8da6-45bc7752c3bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv\" (UID: \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" Oct 30 12:45:32.370032 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.369840 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/da588bf8-ee9a-4103-8da6-45bc7752c3bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv\" (UID: \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" Oct 30 12:45:32.380126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.380045 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"kube-root-ca.crt\"" Oct 30 12:45:32.380126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.380062 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"openshift-service-ca.crt\"" Oct 30 12:45:32.380126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.380105 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0de4b9\"" Oct 30 12:45:32.380439 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.380174 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"routing-overlay\"" Oct 30 12:45:32.384639 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.384528 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"agc-cred-d2104c9f2c1c\"" Oct 30 12:45:32.384639 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.384601 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"agc-cred-d7c4ac25d5ef\"" Oct 30 12:45:32.384878 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.384671 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"payload-processing-external-model-e2e--d115ed06-dockercfg-4k7rq\"" Oct 30 12:45:32.386699 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.386673 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv"] Oct 30 12:45:32.471216 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.471179 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da588bf8-ee9a-4103-8da6-45bc7752c3bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv\" (UID: \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" Oct 30 12:45:32.471458 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.471276 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da588bf8-ee9a-4103-8da6-45bc7752c3bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv\" (UID: \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" Oct 30 12:45:32.471458 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.471373 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/da588bf8-ee9a-4103-8da6-45bc7752c3bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv\" (UID: \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" Oct 30 12:45:32.472426 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.471501 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmlh6\" (UniqueName: \"kubernetes.io/projected/da588bf8-ee9a-4103-8da6-45bc7752c3bb-kube-api-access-hmlh6\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv\" (UID: \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" Oct 30 12:45:32.472426 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.472377 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da588bf8-ee9a-4103-8da6-45bc7752c3bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv\" (UID: \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" Oct 30 12:45:32.472660 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.472588 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da588bf8-ee9a-4103-8da6-45bc7752c3bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv\" (UID: \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" Oct 30 12:45:32.475239 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.475215 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/da588bf8-ee9a-4103-8da6-45bc7752c3bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv\" (UID: \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" Oct 30 12:45:32.501803 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.501771 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmlh6\" (UniqueName: \"kubernetes.io/projected/da588bf8-ee9a-4103-8da6-45bc7752c3bb-kube-api-access-hmlh6\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv\" (UID: \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" Oct 30 12:45:32.507231 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.507202 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:45:32.573008 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.572971 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-podinfo\") pod \"73e2eab8-2195-41e0-97a7-63383ccbbe48\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " Oct 30 12:45:32.573250 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.573026 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-envoy\") pod \"73e2eab8-2195-41e0-97a7-63383ccbbe48\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " Oct 30 12:45:32.573250 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.573212 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-workload-certs\") pod \"73e2eab8-2195-41e0-97a7-63383ccbbe48\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " Oct 30 12:45:32.573393 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.573270 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-credential-socket\") pod \"73e2eab8-2195-41e0-97a7-63383ccbbe48\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " Oct 30 12:45:32.573453 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.573424 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e2eab8-2195-41e0-97a7-63383ccbbe48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e2eab8-2195-41e0-97a7-63383ccbbe48-istiod-ca-cert\") pod \"73e2eab8-2195-41e0-97a7-63383ccbbe48\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " Oct 30 12:45:32.573511 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.573460 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-workload-socket\") pod \"73e2eab8-2195-41e0-97a7-63383ccbbe48\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " Oct 30 12:45:32.573511 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.573496 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-workload-certs" pod "73e2eab8-2195-41e0-97a7-63383ccbbe48" (UID: "73e2eab8-2195-41e0-97a7-63383ccbbe48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:32.573605 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.573511 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-token\" (UniqueName: \"kubernetes.io/projected/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-token\") pod \"73e2eab8-2195-41e0-97a7-63383ccbbe48\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " Oct 30 12:45:32.573605 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.573536 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-credential-socket" pod "73e2eab8-2195-41e0-97a7-63383ccbbe48" (UID: "73e2eab8-2195-41e0-97a7-63383ccbbe48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:32.573605 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.573598 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-data\") pod \"73e2eab8-2195-41e0-97a7-63383ccbbe48\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " Oct 30 12:45:32.573807 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.573651 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e2eab8-2195-41e0-97a7-63383ccbbe48-kube-api-access-4h8v6\" (UniqueName: \"kubernetes.io/projected/73e2eab8-2195-41e0-97a7-63383ccbbe48-kube-api-access-4h8v6\") pod \"73e2eab8-2195-41e0-97a7-63383ccbbe48\" (UID: \"73e2eab8-2195-41e0-97a7-63383ccbbe48\") " Oct 30 12:45:32.573807 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.573679 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-workload-socket" pod "73e2eab8-2195-41e0-97a7-63383ccbbe48" (UID: "73e2eab8-2195-41e0-97a7-63383ccbbe48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:32.573807 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.573787 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e2eab8-2195-41e0-97a7-63383ccbbe48-istiod-ca-cert" pod "73e2eab8-2195-41e0-97a7-63383ccbbe48" (UID: "73e2eab8-2195-41e0-97a7-63383ccbbe48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:32.573973 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.573931 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-data" pod "73e2eab8-2195-41e0-97a7-63383ccbbe48" (UID: "73e2eab8-2195-41e0-97a7-63383ccbbe48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:32.574253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.574234 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.574388 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.574256 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.574388 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.574269 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e2eab8-2195-41e0-97a7-63383ccbbe48-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.574388 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.574281 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.574388 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.574292 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.575751 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.575723 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-token" pod "73e2eab8-2195-41e0-97a7-63383ccbbe48" (UID: "73e2eab8-2195-41e0-97a7-63383ccbbe48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:32.576029 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.575727 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-podinfo" pod "73e2eab8-2195-41e0-97a7-63383ccbbe48" (UID: "73e2eab8-2195-41e0-97a7-63383ccbbe48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:45:32.576029 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.575809 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-envoy" pod "73e2eab8-2195-41e0-97a7-63383ccbbe48" (UID: "73e2eab8-2195-41e0-97a7-63383ccbbe48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:32.576029 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.575873 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e2eab8-2195-41e0-97a7-63383ccbbe48-kube-api-access-4h8v6" pod "73e2eab8-2195-41e0-97a7-63383ccbbe48" (UID: "73e2eab8-2195-41e0-97a7-63383ccbbe48"). InnerVolumeSpecName "kube-api-access-4h8v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:32.673665 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.673628 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" Oct 30 12:45:32.675724 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.675697 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.675801 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.675729 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h8v6\" (UniqueName: \"kubernetes.io/projected/73e2eab8-2195-41e0-97a7-63383ccbbe48-kube-api-access-4h8v6\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.675801 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.675739 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.675801 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.675748 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e2eab8-2195-41e0-97a7-63383ccbbe48-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.995457 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:45:32.995426 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda588bf8_ee9a_4103_8da6_45bc7752c3bb.slice/crio-bc830a31ddd4b3e68a6d3ec468248c2f7bb1c0e93da72891954811b943a48863 WatchSource:0}: Error finding container bc830a31ddd4b3e68a6d3ec468248c2f7bb1c0e93da72891954811b943a48863: Status 404 returned error can't find the container with id bc830a31ddd4b3e68a6d3ec468248c2f7bb1c0e93da72891954811b943a48863 Oct 30 12:45:32.998522 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:32.998498 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv"] Oct 30 12:45:33.329521 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:33.329420 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" event={"ID":"da588bf8-ee9a-4103-8da6-45bc7752c3bb","Type":"ContainerStarted","Data":"85cf4177186dde2fbcbad2946d92425f6db0c71cdbd2e515cecc81f451e00a15"} Oct 30 12:45:33.329521 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:33.329481 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" Oct 30 12:45:33.329521 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:33.329507 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" event={"ID":"da588bf8-ee9a-4103-8da6-45bc7752c3bb","Type":"ContainerStarted","Data":"bc830a31ddd4b3e68a6d3ec468248c2f7bb1c0e93da72891954811b943a48863"} Oct 30 12:45:33.330939 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:33.330922 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" Oct 30 12:45:33.330999 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:33.330957 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd" event={"ID":"73e2eab8-2195-41e0-97a7-63383ccbbe48","Type":"ContainerDied","Data":"7953dfb24fea2c27bfffeb03175cb1054a6bc499bbde997232a5da4f13f7d043"} Oct 30 12:45:33.331056 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:33.331004 4468 scope.go:122] "RemoveContainer" containerID="c5c50ce3d3ce81f5687ab0de35881a9ca9075703802cc431f0f0bf62127e30c4" Oct 30 12:45:33.385469 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:33.385406 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" podStartSLOduration=1.385390052 podStartE2EDuration="1.385390052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:33.383848706 +0000 UTC m=+3084.417987102" watchObservedRunningTime="2026-10-30 12:45:33.385390052 +0000 UTC m=+3084.419528448" Oct 30 12:45:33.438450 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:33.438419 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd"] Oct 30 12:45:33.450620 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:33.450590 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-rflkd"] Oct 30 12:45:35.094703 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.094662 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e2eab8-2195-41e0-97a7-63383ccbbe48" path="/var/lib/kubelet/pods/73e2eab8-2195-41e0-97a7-63383ccbbe48/volumes" Oct 30 12:45:35.106822 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.106796 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:45:35.219679 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.218578 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istiod-ca-cert" pod "b7abde5f-37e2-40d0-9ddc-bd907e115aa0" (UID: "b7abde5f-37e2-40d0-9ddc-bd907e115aa0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:35.219679 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.218168 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istiod-ca-cert\") pod \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " Oct 30 12:45:35.219679 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.218794 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-credential-socket\") pod \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " Oct 30 12:45:35.219679 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.219116 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-credential-socket" pod "b7abde5f-37e2-40d0-9ddc-bd907e115aa0" (UID: "b7abde5f-37e2-40d0-9ddc-bd907e115aa0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:35.219679 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.219163 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-workload-socket\") pod \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " Oct 30 12:45:35.219679 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.219197 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-token\" (UniqueName: \"kubernetes.io/projected/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-token\") pod \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " Oct 30 12:45:35.219679 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.219233 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-workload-certs\") pod \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " Oct 30 12:45:35.219679 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.219365 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-kube-api-access-vtkx6\" (UniqueName: \"kubernetes.io/projected/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-kube-api-access-vtkx6\") pod \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " Oct 30 12:45:35.219679 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.219458 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-podinfo\") pod \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " Oct 30 12:45:35.219679 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.219487 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-data\") pod \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " Oct 30 12:45:35.219679 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.219558 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-envoy\") pod \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\" (UID: \"b7abde5f-37e2-40d0-9ddc-bd907e115aa0\") " Oct 30 12:45:35.220506 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.220135 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:35.220506 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.220152 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:35.220631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.220520 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-workload-certs" pod "b7abde5f-37e2-40d0-9ddc-bd907e115aa0" (UID: "b7abde5f-37e2-40d0-9ddc-bd907e115aa0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:35.221021 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.220925 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-workload-socket" pod "b7abde5f-37e2-40d0-9ddc-bd907e115aa0" (UID: "b7abde5f-37e2-40d0-9ddc-bd907e115aa0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:35.221166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.221048 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-data" pod "b7abde5f-37e2-40d0-9ddc-bd907e115aa0" (UID: "b7abde5f-37e2-40d0-9ddc-bd907e115aa0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:35.222677 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.222645 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-envoy" pod "b7abde5f-37e2-40d0-9ddc-bd907e115aa0" (UID: "b7abde5f-37e2-40d0-9ddc-bd907e115aa0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:35.223177 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.223151 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-podinfo" pod "b7abde5f-37e2-40d0-9ddc-bd907e115aa0" (UID: "b7abde5f-37e2-40d0-9ddc-bd907e115aa0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:45:35.223257 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.223227 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-token" pod "b7abde5f-37e2-40d0-9ddc-bd907e115aa0" (UID: "b7abde5f-37e2-40d0-9ddc-bd907e115aa0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:35.223452 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.223427 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-kube-api-access-vtkx6" pod "b7abde5f-37e2-40d0-9ddc-bd907e115aa0" (UID: "b7abde5f-37e2-40d0-9ddc-bd907e115aa0"). InnerVolumeSpecName "kube-api-access-vtkx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:35.321463 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.321309 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:35.321463 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.321347 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:35.321463 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.321360 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:35.321463 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.321373 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:35.321463 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.321385 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:35.321463 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.321399 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:35.321463 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.321411 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtkx6\" (UniqueName: \"kubernetes.io/projected/b7abde5f-37e2-40d0-9ddc-bd907e115aa0-kube-api-access-vtkx6\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:35.348286 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.348249 4468 generic.go:356] "Generic (PLEG): container finished" podID="b7abde5f-37e2-40d0-9ddc-bd907e115aa0" containerID="a50e1660fc0db064e39bfea6e4a5c6743762d77870e738cf16ad987d9cf4f198" exitCode=0 Oct 30 12:45:35.348457 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.348302 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" Oct 30 12:45:35.348457 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.348329 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" event={"ID":"b7abde5f-37e2-40d0-9ddc-bd907e115aa0","Type":"ContainerDied","Data":"a50e1660fc0db064e39bfea6e4a5c6743762d77870e738cf16ad987d9cf4f198"} Oct 30 12:45:35.348457 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.348387 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt" event={"ID":"b7abde5f-37e2-40d0-9ddc-bd907e115aa0","Type":"ContainerDied","Data":"848334bf89b56fe026b8fca853a15b3ec190cfcb34b071434a15fb0a0b05b981"} Oct 30 12:45:35.348457 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.348415 4468 scope.go:122] "RemoveContainer" containerID="a50e1660fc0db064e39bfea6e4a5c6743762d77870e738cf16ad987d9cf4f198" Oct 30 12:45:35.364157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.364132 4468 scope.go:122] "RemoveContainer" containerID="a50e1660fc0db064e39bfea6e4a5c6743762d77870e738cf16ad987d9cf4f198" Oct 30 12:45:35.364544 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:45:35.364521 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50e1660fc0db064e39bfea6e4a5c6743762d77870e738cf16ad987d9cf4f198\": container with ID starting with a50e1660fc0db064e39bfea6e4a5c6743762d77870e738cf16ad987d9cf4f198 not found: ID does not exist" containerID="a50e1660fc0db064e39bfea6e4a5c6743762d77870e738cf16ad987d9cf4f198" Oct 30 12:45:35.364602 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.364554 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50e1660fc0db064e39bfea6e4a5c6743762d77870e738cf16ad987d9cf4f198"} err="failed to get container status \"a50e1660fc0db064e39bfea6e4a5c6743762d77870e738cf16ad987d9cf4f198\": rpc error: code = NotFound desc = could not find container \"a50e1660fc0db064e39bfea6e4a5c6743762d77870e738cf16ad987d9cf4f198\": container with ID starting with a50e1660fc0db064e39bfea6e4a5c6743762d77870e738cf16ad987d9cf4f198 not found: ID does not exist" Oct 30 12:45:35.417707 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.417665 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt"] Oct 30 12:45:35.422661 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:35.422633 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-4rpkt"] Oct 30 12:45:37.093702 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:37.093668 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7abde5f-37e2-40d0-9ddc-bd907e115aa0" path="/var/lib/kubelet/pods/b7abde5f-37e2-40d0-9ddc-bd907e115aa0/volumes" Oct 30 12:45:39.874646 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:39.874601 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww"] Oct 30 12:45:44.342960 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:44.342928 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" Oct 30 12:45:44.874989 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:44.874941 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" podUID="af776420-bc69-4979-9d77-ac5c7d22c2a4" containerName="payload-processing" containerID="cri-o://c65b09051e4fd66920578813d4c5797fa678628b0a5f65950e7f0cfbc80e2f8f" gracePeriod=25 Oct 30 12:45:45.162307 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.162276 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" Oct 30 12:45:45.270968 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.270929 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af776420-bc69-4979-9d77-ac5c7d22c2a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af776420-bc69-4979-9d77-ac5c7d22c2a4-plugins-config-volume\") pod \"af776420-bc69-4979-9d77-ac5c7d22c2a4\" (UID: \"af776420-bc69-4979-9d77-ac5c7d22c2a4\") " Oct 30 12:45:45.271181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.271025 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af776420-bc69-4979-9d77-ac5c7d22c2a4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af776420-bc69-4979-9d77-ac5c7d22c2a4-routing-overlay-volume\") pod \"af776420-bc69-4979-9d77-ac5c7d22c2a4\" (UID: \"af776420-bc69-4979-9d77-ac5c7d22c2a4\") " Oct 30 12:45:45.271181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.271048 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af776420-bc69-4979-9d77-ac5c7d22c2a4-kube-api-access-vffb9\" (UniqueName: \"kubernetes.io/projected/af776420-bc69-4979-9d77-ac5c7d22c2a4-kube-api-access-vffb9\") pod \"af776420-bc69-4979-9d77-ac5c7d22c2a4\" (UID: \"af776420-bc69-4979-9d77-ac5c7d22c2a4\") " Oct 30 12:45:45.271181 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.271171 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af776420-bc69-4979-9d77-ac5c7d22c2a4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/af776420-bc69-4979-9d77-ac5c7d22c2a4-provider-credentials\") pod \"af776420-bc69-4979-9d77-ac5c7d22c2a4\" (UID: \"af776420-bc69-4979-9d77-ac5c7d22c2a4\") " Oct 30 12:45:45.271409 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.271387 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af776420-bc69-4979-9d77-ac5c7d22c2a4-plugins-config-volume" pod "af776420-bc69-4979-9d77-ac5c7d22c2a4" (UID: "af776420-bc69-4979-9d77-ac5c7d22c2a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:45.271461 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.271399 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af776420-bc69-4979-9d77-ac5c7d22c2a4-routing-overlay-volume" pod "af776420-bc69-4979-9d77-ac5c7d22c2a4" (UID: "af776420-bc69-4979-9d77-ac5c7d22c2a4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:45.271688 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.271673 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af776420-bc69-4979-9d77-ac5c7d22c2a4-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:45.271746 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.271692 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af776420-bc69-4979-9d77-ac5c7d22c2a4-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:45.273469 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.273448 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af776420-bc69-4979-9d77-ac5c7d22c2a4-kube-api-access-vffb9" pod "af776420-bc69-4979-9d77-ac5c7d22c2a4" (UID: "af776420-bc69-4979-9d77-ac5c7d22c2a4"). InnerVolumeSpecName "kube-api-access-vffb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:45.273532 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.273481 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af776420-bc69-4979-9d77-ac5c7d22c2a4-provider-credentials" pod "af776420-bc69-4979-9d77-ac5c7d22c2a4" (UID: "af776420-bc69-4979-9d77-ac5c7d22c2a4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:45.373496 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.373448 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af776420-bc69-4979-9d77-ac5c7d22c2a4-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:45.373496 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.373488 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vffb9\" (UniqueName: \"kubernetes.io/projected/af776420-bc69-4979-9d77-ac5c7d22c2a4-kube-api-access-vffb9\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:45:45.444605 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.444572 4468 generic.go:356] "Generic (PLEG): container finished" podID="af776420-bc69-4979-9d77-ac5c7d22c2a4" containerID="c65b09051e4fd66920578813d4c5797fa678628b0a5f65950e7f0cfbc80e2f8f" exitCode=0 Oct 30 12:45:45.444827 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.444627 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" Oct 30 12:45:45.444827 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.444655 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" event={"ID":"af776420-bc69-4979-9d77-ac5c7d22c2a4","Type":"ContainerDied","Data":"c65b09051e4fd66920578813d4c5797fa678628b0a5f65950e7f0cfbc80e2f8f"} Oct 30 12:45:45.444827 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.444716 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww" event={"ID":"af776420-bc69-4979-9d77-ac5c7d22c2a4","Type":"ContainerDied","Data":"2ec4c0b99e9b2aca22bd3ba5409cd71eb435facd52a45ab49bb495319aea6aba"} Oct 30 12:45:45.444827 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.444746 4468 scope.go:122] "RemoveContainer" containerID="c65b09051e4fd66920578813d4c5797fa678628b0a5f65950e7f0cfbc80e2f8f" Oct 30 12:45:45.456471 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.456447 4468 scope.go:122] "RemoveContainer" containerID="c65b09051e4fd66920578813d4c5797fa678628b0a5f65950e7f0cfbc80e2f8f" Oct 30 12:45:45.456753 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:45:45.456735 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c65b09051e4fd66920578813d4c5797fa678628b0a5f65950e7f0cfbc80e2f8f\": container with ID starting with c65b09051e4fd66920578813d4c5797fa678628b0a5f65950e7f0cfbc80e2f8f not found: ID does not exist" containerID="c65b09051e4fd66920578813d4c5797fa678628b0a5f65950e7f0cfbc80e2f8f" Oct 30 12:45:45.456816 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.456763 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c65b09051e4fd66920578813d4c5797fa678628b0a5f65950e7f0cfbc80e2f8f"} err="failed to get container status \"c65b09051e4fd66920578813d4c5797fa678628b0a5f65950e7f0cfbc80e2f8f\": rpc error: code = NotFound desc = could not find container \"c65b09051e4fd66920578813d4c5797fa678628b0a5f65950e7f0cfbc80e2f8f\": container with ID starting with c65b09051e4fd66920578813d4c5797fa678628b0a5f65950e7f0cfbc80e2f8f not found: ID does not exist" Oct 30 12:45:45.503582 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.503541 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww"] Oct 30 12:45:45.517688 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:45.517652 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-566fkcvww"] Oct 30 12:45:47.093996 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:47.093951 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af776420-bc69-4979-9d77-ac5c7d22c2a4" path="/var/lib/kubelet/pods/af776420-bc69-4979-9d77-ac5c7d22c2a4/volumes" Oct 30 12:45:53.601522 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:45:53.601439 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:45:56.709174 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.709134 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd"] Oct 30 12:45:56.709882 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.709868 4468 cpu_manager.go:414] "containerMap: removing container" podUID="73e2eab8-2195-41e0-97a7-63383ccbbe48" containerName="istio-proxy" Oct 30 12:45:56.709923 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.709883 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e2eab8-2195-41e0-97a7-63383ccbbe48" containerName="istio-proxy" Oct 30 12:45:56.709923 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.709907 4468 cpu_manager.go:414] "containerMap: removing container" podUID="1f58a672-ac8b-4715-aa9a-b9d16de76312" containerName="cleanup" Oct 30 12:45:56.709923 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.709913 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f58a672-ac8b-4715-aa9a-b9d16de76312" containerName="cleanup" Oct 30 12:45:56.709923 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.709923 4468 cpu_manager.go:414] "containerMap: removing container" podUID="b7abde5f-37e2-40d0-9ddc-bd907e115aa0" containerName="istio-proxy" Oct 30 12:45:56.710061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.709929 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7abde5f-37e2-40d0-9ddc-bd907e115aa0" containerName="istio-proxy" Oct 30 12:45:56.710061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.709938 4468 cpu_manager.go:414] "containerMap: removing container" podUID="73e2eab8-2195-41e0-97a7-63383ccbbe48" containerName="istio-proxy" Oct 30 12:45:56.710061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.709942 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e2eab8-2195-41e0-97a7-63383ccbbe48" containerName="istio-proxy" Oct 30 12:45:56.710061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.709961 4468 cpu_manager.go:414] "containerMap: removing container" podUID="aafad9d8-210e-4385-8d3a-447fa25f1d53" containerName="cleanup" Oct 30 12:45:56.710061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.709966 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafad9d8-210e-4385-8d3a-447fa25f1d53" containerName="cleanup" Oct 30 12:45:56.710061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.709976 4468 cpu_manager.go:414] "containerMap: removing container" podUID="af776420-bc69-4979-9d77-ac5c7d22c2a4" containerName="payload-processing" Oct 30 12:45:56.710061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.709980 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af776420-bc69-4979-9d77-ac5c7d22c2a4" containerName="payload-processing" Oct 30 12:45:56.710299 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.710097 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b12758d-18f5-46f4-83b7-c52251978743" containerName="cleanup" Oct 30 12:45:56.710299 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.710108 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e2eab8-2195-41e0-97a7-63383ccbbe48" containerName="istio-proxy" Oct 30 12:45:56.710299 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.710117 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="aafad9d8-210e-4385-8d3a-447fa25f1d53" containerName="cleanup" Oct 30 12:45:56.710299 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.710124 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f58a672-ac8b-4715-aa9a-b9d16de76312" containerName="cleanup" Oct 30 12:45:56.710299 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.710132 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7abde5f-37e2-40d0-9ddc-bd907e115aa0" containerName="istio-proxy" Oct 30 12:45:56.710299 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.710141 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="af776420-bc69-4979-9d77-ac5c7d22c2a4" containerName="payload-processing" Oct 30 12:45:56.715391 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.715364 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" Oct 30 12:45:56.748170 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.748133 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"e2e-cmp-legacy-f22f40-provider-cred\"" Oct 30 12:45:56.748400 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.748217 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"kube-root-ca.crt\"" Oct 30 12:45:56.748509 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.748303 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"routing-overlay\"" Oct 30 12:45:56.748601 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.748328 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"payload-processing-external-model-e2e--03fcd75d-dockercfg-bt8n4\"" Oct 30 12:45:56.758739 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.758706 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd"] Oct 30 12:45:56.781462 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.781423 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/748d03d3-bf28-4060-8821-8b558864dafb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-788562khd\" (UID: \"748d03d3-bf28-4060-8821-8b558864dafb\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" Oct 30 12:45:56.781628 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.781530 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/748d03d3-bf28-4060-8821-8b558864dafb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-788562khd\" (UID: \"748d03d3-bf28-4060-8821-8b558864dafb\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" Oct 30 12:45:56.781628 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.781581 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/748d03d3-bf28-4060-8821-8b558864dafb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-788562khd\" (UID: \"748d03d3-bf28-4060-8821-8b558864dafb\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" Oct 30 12:45:56.781747 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.781647 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b92d9\" (UniqueName: \"kubernetes.io/projected/748d03d3-bf28-4060-8821-8b558864dafb-kube-api-access-b92d9\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-788562khd\" (UID: \"748d03d3-bf28-4060-8821-8b558864dafb\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" Oct 30 12:45:56.850943 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.848680 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2545c7\"" Oct 30 12:45:56.879263 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.879217 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"openshift-service-ca.crt\"" Oct 30 12:45:56.906203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.894874 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/748d03d3-bf28-4060-8821-8b558864dafb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-788562khd\" (UID: \"748d03d3-bf28-4060-8821-8b558864dafb\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" Oct 30 12:45:56.906203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.894961 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/748d03d3-bf28-4060-8821-8b558864dafb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-788562khd\" (UID: \"748d03d3-bf28-4060-8821-8b558864dafb\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" Oct 30 12:45:56.906203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.895047 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b92d9\" (UniqueName: \"kubernetes.io/projected/748d03d3-bf28-4060-8821-8b558864dafb-kube-api-access-b92d9\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-788562khd\" (UID: \"748d03d3-bf28-4060-8821-8b558864dafb\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" Oct 30 12:45:56.906203 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.895267 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/748d03d3-bf28-4060-8821-8b558864dafb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-788562khd\" (UID: \"748d03d3-bf28-4060-8821-8b558864dafb\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" Oct 30 12:45:56.913421 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.908097 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/748d03d3-bf28-4060-8821-8b558864dafb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-788562khd\" (UID: \"748d03d3-bf28-4060-8821-8b558864dafb\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" Oct 30 12:45:56.916100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.916041 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/748d03d3-bf28-4060-8821-8b558864dafb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-788562khd\" (UID: \"748d03d3-bf28-4060-8821-8b558864dafb\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" Oct 30 12:45:56.926230 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.926195 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/748d03d3-bf28-4060-8821-8b558864dafb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-788562khd\" (UID: \"748d03d3-bf28-4060-8821-8b558864dafb\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" Oct 30 12:45:56.943880 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:56.943842 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b92d9\" (UniqueName: \"kubernetes.io/projected/748d03d3-bf28-4060-8821-8b558864dafb-kube-api-access-b92d9\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-788562khd\" (UID: \"748d03d3-bf28-4060-8821-8b558864dafb\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" Oct 30 12:45:57.031213 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:57.031113 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" Oct 30 12:45:57.359855 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:45:57.359811 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod748d03d3_bf28_4060_8821_8b558864dafb.slice/crio-bb2dbd0dae513ecc825dbde954bf6fb24fcd77f877af64fcaf749049d286264a WatchSource:0}: Error finding container bb2dbd0dae513ecc825dbde954bf6fb24fcd77f877af64fcaf749049d286264a: Status 404 returned error can't find the container with id bb2dbd0dae513ecc825dbde954bf6fb24fcd77f877af64fcaf749049d286264a Oct 30 12:45:57.363485 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:57.363459 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd"] Oct 30 12:45:57.580283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:57.579474 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" event={"ID":"748d03d3-bf28-4060-8821-8b558864dafb","Type":"ContainerStarted","Data":"4eba19738e5d7c295e5a0ff7c4d1481086c3aa0c4c48dc4ba4ac80ddeb472ae4"} Oct 30 12:45:57.580283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:57.579544 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" event={"ID":"748d03d3-bf28-4060-8821-8b558864dafb","Type":"ContainerStarted","Data":"bb2dbd0dae513ecc825dbde954bf6fb24fcd77f877af64fcaf749049d286264a"} Oct 30 12:45:57.580283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:57.579648 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" Oct 30 12:45:57.634551 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:57.634443 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" podStartSLOduration=1.634428491 podStartE2EDuration="1.634428491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:57.630902402 +0000 UTC m=+3108.665040797" watchObservedRunningTime="2026-10-30 12:45:57.634428491 +0000 UTC m=+3108.668566926" Oct 30 12:45:58.883959 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:58.883918 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2"] Oct 30 12:45:58.884848 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:58.884831 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e2eab8-2195-41e0-97a7-63383ccbbe48" containerName="istio-proxy" Oct 30 12:45:58.888891 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:58.888870 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" Oct 30 12:45:58.910264 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:58.910233 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2"] Oct 30 12:45:58.916766 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:58.916679 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"e2e-cmp-legacy-d77a85-provider-cred\"" Oct 30 12:45:58.916766 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:58.916752 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a786af\"" Oct 30 12:45:58.917061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:58.916953 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"kube-root-ca.crt\"" Oct 30 12:45:58.917061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:58.916956 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"openshift-service-ca.crt\"" Oct 30 12:45:58.917217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:58.917192 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"routing-overlay\"" Oct 30 12:45:58.920129 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:58.920105 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"payload-processing-external-model-e2e--3f99fe6c-dockercfg-rwchz\"" Oct 30 12:45:59.050189 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:59.050093 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2\" (UID: \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" Oct 30 12:45:59.050386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:59.050301 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xpgw\" (UniqueName: \"kubernetes.io/projected/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-kube-api-access-6xpgw\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2\" (UID: \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" Oct 30 12:45:59.050386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:59.050347 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2\" (UID: \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" Oct 30 12:45:59.050386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:59.050371 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2\" (UID: \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" Oct 30 12:45:59.151424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:59.151335 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpgw\" (UniqueName: \"kubernetes.io/projected/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-kube-api-access-6xpgw\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2\" (UID: \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" Oct 30 12:45:59.151424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:59.151377 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2\" (UID: \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" Oct 30 12:45:59.151424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:59.151397 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2\" (UID: \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" Oct 30 12:45:59.151722 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:59.151513 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2\" (UID: \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" Oct 30 12:45:59.152021 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:59.151998 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2\" (UID: \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" Oct 30 12:45:59.152174 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:59.152156 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2\" (UID: \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" Oct 30 12:45:59.154253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:59.154217 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2\" (UID: \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" Oct 30 12:45:59.177780 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:59.177733 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xpgw\" (UniqueName: \"kubernetes.io/projected/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-kube-api-access-6xpgw\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2\" (UID: \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" Oct 30 12:45:59.205467 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:59.205433 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" Oct 30 12:45:59.520701 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:45:59.520666 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce74d0e0_2e5a_4213_a460_7430ca8f3cb7.slice/crio-6021fb4ed0915a3b855a30cccc3879526fd03755748911f0bc6bb5aeddc16dee WatchSource:0}: Error finding container 6021fb4ed0915a3b855a30cccc3879526fd03755748911f0bc6bb5aeddc16dee: Status 404 returned error can't find the container with id 6021fb4ed0915a3b855a30cccc3879526fd03755748911f0bc6bb5aeddc16dee Oct 30 12:45:59.523899 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:59.523869 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2"] Oct 30 12:45:59.597733 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:59.597694 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" event={"ID":"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7","Type":"ContainerStarted","Data":"1b9818f202e732518f56b73c09de93cf8d5f355a0af0c7125f2ef31e2e8feac8"} Oct 30 12:45:59.597924 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:59.597762 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" event={"ID":"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7","Type":"ContainerStarted","Data":"6021fb4ed0915a3b855a30cccc3879526fd03755748911f0bc6bb5aeddc16dee"} Oct 30 12:45:59.597924 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:59.597808 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" Oct 30 12:45:59.660617 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:45:59.660405 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" podStartSLOduration=1.6603876290000001 podStartE2EDuration="1.660387629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:59.654961302 +0000 UTC m=+3110.689099698" watchObservedRunningTime="2026-10-30 12:45:59.660387629 +0000 UTC m=+3110.694526026" Oct 30 12:46:08.592379 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:08.592349 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" Oct 30 12:46:10.611117 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:10.611050 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" Oct 30 12:46:23.401333 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.401294 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm"] Oct 30 12:46:23.407961 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.407930 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" Oct 30 12:46:23.425253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.425219 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm"] Oct 30 12:46:23.434126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.433457 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"payload-processing-external-model-e2e--a9926bc2-dockercfg-fx898\"" Oct 30 12:46:23.434126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.433519 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"openshift-service-ca.crt\"" Oct 30 12:46:23.434126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.433540 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"agc-cred-3cd06b9a0e1c\"" Oct 30 12:46:23.434126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.433568 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"routing-overlay\"" Oct 30 12:46:23.434126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.433559 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-f7464f\"" Oct 30 12:46:23.438007 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.437986 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"kube-root-ca.crt\"" Oct 30 12:46:23.554426 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.554382 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdxc2\" (UniqueName: \"kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-kube-api-access-jdxc2\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm\" (UID: \"198e65ad-90a3-44de-9576-ff24d03b805e\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" Oct 30 12:46:23.554699 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.554669 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198e65ad-90a3-44de-9576-ff24d03b805e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm\" (UID: \"198e65ad-90a3-44de-9576-ff24d03b805e\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" Oct 30 12:46:23.554859 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.554728 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/198e65ad-90a3-44de-9576-ff24d03b805e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm\" (UID: \"198e65ad-90a3-44de-9576-ff24d03b805e\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" Oct 30 12:46:23.554859 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.554764 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm\" (UID: \"198e65ad-90a3-44de-9576-ff24d03b805e\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" Oct 30 12:46:23.603491 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:46:23.603456 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:46:23.656254 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.656160 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198e65ad-90a3-44de-9576-ff24d03b805e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm\" (UID: \"198e65ad-90a3-44de-9576-ff24d03b805e\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" Oct 30 12:46:23.656254 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.656212 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/198e65ad-90a3-44de-9576-ff24d03b805e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm\" (UID: \"198e65ad-90a3-44de-9576-ff24d03b805e\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" Oct 30 12:46:23.656254 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.656239 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm\" (UID: \"198e65ad-90a3-44de-9576-ff24d03b805e\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" Oct 30 12:46:23.656490 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.656270 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdxc2\" (UniqueName: \"kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-kube-api-access-jdxc2\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm\" (UID: \"198e65ad-90a3-44de-9576-ff24d03b805e\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" Oct 30 12:46:23.656490 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:46:23.656403 4468 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f7464f/agc-cred-3cd06b9a0e1c: secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:46:23.656490 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:46:23.656425 4468 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm: secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:46:23.656608 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:46:23.656495 4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-provider-credentials podName:198e65ad-90a3-44de-9576-ff24d03b805e nodeName:}" failed. No retries permitted until 2026-10-30 12:46:24.156480023 +0000 UTC m=+3135.190618410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" (UID: "198e65ad-90a3-44de-9576-ff24d03b805e") : secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:46:23.657065 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.657035 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/198e65ad-90a3-44de-9576-ff24d03b805e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm\" (UID: \"198e65ad-90a3-44de-9576-ff24d03b805e\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" Oct 30 12:46:23.657232 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.657213 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198e65ad-90a3-44de-9576-ff24d03b805e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm\" (UID: \"198e65ad-90a3-44de-9576-ff24d03b805e\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" Oct 30 12:46:23.683270 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:23.683222 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdxc2\" (UniqueName: \"kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-kube-api-access-jdxc2\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm\" (UID: \"198e65ad-90a3-44de-9576-ff24d03b805e\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" Oct 30 12:46:24.167234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:24.167192 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm\" (UID: \"198e65ad-90a3-44de-9576-ff24d03b805e\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" Oct 30 12:46:24.167469 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:46:24.167340 4468 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f7464f/agc-cred-3cd06b9a0e1c: secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:46:24.167469 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:46:24.167361 4468 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm: secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:46:24.167469 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:46:24.167416 4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-provider-credentials podName:198e65ad-90a3-44de-9576-ff24d03b805e nodeName:}" failed. No retries permitted until 2026-10-30 12:46:25.167402363 +0000 UTC m=+3136.201540750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" (UID: "198e65ad-90a3-44de-9576-ff24d03b805e") : secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:46:24.588800 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:24.588695 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv"] Oct 30 12:46:24.593577 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:24.593548 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" Oct 30 12:46:24.614580 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:24.614540 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv"] Oct 30 12:46:24.614966 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:24.614941 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"e2e-cfl-69b234-provider-cred\"" Oct 30 12:46:24.674439 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:24.674402 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv\" (UID: \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" Oct 30 12:46:24.674601 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:24.674559 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv\" (UID: \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" Oct 30 12:46:24.674667 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:24.674596 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pswz6\" (UniqueName: \"kubernetes.io/projected/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-kube-api-access-pswz6\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv\" (UID: \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" Oct 30 12:46:24.674706 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:24.674679 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv\" (UID: \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" Oct 30 12:46:24.775780 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:24.775742 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv\" (UID: \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" Oct 30 12:46:24.775991 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:24.775911 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv\" (UID: \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" Oct 30 12:46:24.776285 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:24.776030 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv\" (UID: \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" Oct 30 12:46:24.776285 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:24.776108 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pswz6\" (UniqueName: \"kubernetes.io/projected/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-kube-api-access-pswz6\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv\" (UID: \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" Oct 30 12:46:24.776608 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:24.776585 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv\" (UID: \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" Oct 30 12:46:24.776729 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:24.776705 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv\" (UID: \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" Oct 30 12:46:24.778634 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:24.778601 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv\" (UID: \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" Oct 30 12:46:24.801447 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:24.801403 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pswz6\" (UniqueName: \"kubernetes.io/projected/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-kube-api-access-pswz6\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv\" (UID: \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" Oct 30 12:46:24.912882 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:24.912836 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" Oct 30 12:46:25.196620 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:25.196518 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm\" (UID: \"198e65ad-90a3-44de-9576-ff24d03b805e\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" Oct 30 12:46:25.196816 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:46:25.196686 4468 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f7464f/agc-cred-3cd06b9a0e1c: secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:46:25.196816 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:46:25.196709 4468 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm: secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:46:25.196816 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:46:25.196767 4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-provider-credentials podName:198e65ad-90a3-44de-9576-ff24d03b805e nodeName:}" failed. No retries permitted until 2026-10-30 12:46:27.196746912 +0000 UTC m=+3138.230885308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" (UID: "198e65ad-90a3-44de-9576-ff24d03b805e") : secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:46:25.225318 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:46:25.225279 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea3ef2f_83dc_49fb_ad52_c02fdbdfde00.slice/crio-6a98b8319d5749ca9791e28b040a7f9f31c888dedcc33b9fa6b6d134a69104f8 WatchSource:0}: Error finding container 6a98b8319d5749ca9791e28b040a7f9f31c888dedcc33b9fa6b6d134a69104f8: Status 404 returned error can't find the container with id 6a98b8319d5749ca9791e28b040a7f9f31c888dedcc33b9fa6b6d134a69104f8 Oct 30 12:46:25.229607 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:25.229577 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv"] Oct 30 12:46:25.808142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:25.808102 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" event={"ID":"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00","Type":"ContainerStarted","Data":"bae2ceadeee628c3da281d51085ded362239cb36cd2e248a1d1dcb0cd4f38c60"} Oct 30 12:46:25.808633 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:25.808164 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" event={"ID":"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00","Type":"ContainerStarted","Data":"6a98b8319d5749ca9791e28b040a7f9f31c888dedcc33b9fa6b6d134a69104f8"} Oct 30 12:46:25.808633 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:25.808200 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" Oct 30 12:46:25.874367 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:25.874309 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" podStartSLOduration=1.874290009 podStartE2EDuration="1.874290009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:25.870647321 +0000 UTC m=+3136.904785718" watchObservedRunningTime="2026-10-30 12:46:25.874290009 +0000 UTC m=+3136.908428405" Oct 30 12:46:27.227869 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:27.227828 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm\" (UID: \"198e65ad-90a3-44de-9576-ff24d03b805e\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" Oct 30 12:46:27.228303 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:46:27.227943 4468 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f7464f/agc-cred-3cd06b9a0e1c: secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:46:27.228303 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:46:27.227965 4468 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm: secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:46:27.228303 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:46:27.228025 4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-provider-credentials podName:198e65ad-90a3-44de-9576-ff24d03b805e nodeName:}" failed. No retries permitted until 2026-10-30 12:46:31.228010542 +0000 UTC m=+3142.262148916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" (UID: "198e65ad-90a3-44de-9576-ff24d03b805e") : secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:46:31.295548 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:31.295506 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm\" (UID: \"198e65ad-90a3-44de-9576-ff24d03b805e\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" Oct 30 12:46:31.295977 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:46:31.295662 4468 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f7464f/agc-cred-3cd06b9a0e1c: secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:46:31.295977 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:46:31.295683 4468 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm: secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:46:31.295977 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:46:31.295760 4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-provider-credentials podName:198e65ad-90a3-44de-9576-ff24d03b805e nodeName:}" failed. No retries permitted until 2026-10-30 12:46:39.295745207 +0000 UTC m=+3150.329883594 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" (UID: "198e65ad-90a3-44de-9576-ff24d03b805e") : secret "agc-cred-3cd06b9a0e1c" not found Oct 30 12:46:36.820329 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:36.820289 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" Oct 30 12:46:36.925283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:36.925232 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm"] Oct 30 12:46:36.926378 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:46:36.926339 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" podUID="198e65ad-90a3-44de-9576-ff24d03b805e" Oct 30 12:46:37.901463 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:37.901422 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" Oct 30 12:46:37.913861 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:37.913827 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" Oct 30 12:46:38.005194 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:38.005146 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/198e65ad-90a3-44de-9576-ff24d03b805e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/198e65ad-90a3-44de-9576-ff24d03b805e-routing-overlay-volume\") pod \"198e65ad-90a3-44de-9576-ff24d03b805e\" (UID: \"198e65ad-90a3-44de-9576-ff24d03b805e\") " Oct 30 12:46:38.005413 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:38.005292 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/198e65ad-90a3-44de-9576-ff24d03b805e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198e65ad-90a3-44de-9576-ff24d03b805e-plugins-config-volume\") pod \"198e65ad-90a3-44de-9576-ff24d03b805e\" (UID: \"198e65ad-90a3-44de-9576-ff24d03b805e\") " Oct 30 12:46:38.005413 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:38.005393 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-kube-api-access-jdxc2\" (UniqueName: \"kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-kube-api-access-jdxc2\") pod \"198e65ad-90a3-44de-9576-ff24d03b805e\" (UID: \"198e65ad-90a3-44de-9576-ff24d03b805e\") " Oct 30 12:46:38.005627 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:38.005599 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/198e65ad-90a3-44de-9576-ff24d03b805e-routing-overlay-volume" pod "198e65ad-90a3-44de-9576-ff24d03b805e" (UID: "198e65ad-90a3-44de-9576-ff24d03b805e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:46:38.005762 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:38.005736 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/198e65ad-90a3-44de-9576-ff24d03b805e-plugins-config-volume" pod "198e65ad-90a3-44de-9576-ff24d03b805e" (UID: "198e65ad-90a3-44de-9576-ff24d03b805e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:46:38.006010 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:38.005992 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/198e65ad-90a3-44de-9576-ff24d03b805e-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:46:38.006094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:38.006017 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198e65ad-90a3-44de-9576-ff24d03b805e-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:46:38.007737 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:38.007709 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-kube-api-access-jdxc2" pod "198e65ad-90a3-44de-9576-ff24d03b805e" (UID: "198e65ad-90a3-44de-9576-ff24d03b805e"). InnerVolumeSpecName "kube-api-access-jdxc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:46:38.107257 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:38.107216 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdxc2\" (UniqueName: \"kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-kube-api-access-jdxc2\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:46:38.908884 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:38.908837 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm" Oct 30 12:46:39.015127 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:39.011042 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm"] Oct 30 12:46:39.018812 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:39.018776 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-694425pfm"] Oct 30 12:46:39.093126 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:39.093091 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="198e65ad-90a3-44de-9576-ff24d03b805e" path="/var/lib/kubelet/pods/198e65ad-90a3-44de-9576-ff24d03b805e/volumes" Oct 30 12:46:39.121798 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:39.121757 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/198e65ad-90a3-44de-9576-ff24d03b805e-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:46:53.602211 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:46:53.602173 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:46:57.330123 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:57.330081 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c"] Oct 30 12:46:57.335185 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:57.335158 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" Oct 30 12:46:57.353934 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:57.353896 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c"] Oct 30 12:46:57.361283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:57.361237 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"agc-cred-df586a16fd3d\"" Oct 30 12:46:57.411319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:57.411272 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c\" (UID: \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" Oct 30 12:46:57.411594 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:57.411361 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vfql\" (UniqueName: \"kubernetes.io/projected/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-kube-api-access-7vfql\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c\" (UID: \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" Oct 30 12:46:57.411594 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:57.411438 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c\" (UID: \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" Oct 30 12:46:57.411594 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:57.411473 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c\" (UID: \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" Oct 30 12:46:57.512959 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:57.512919 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c\" (UID: \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" Oct 30 12:46:57.513195 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:57.512982 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vfql\" (UniqueName: \"kubernetes.io/projected/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-kube-api-access-7vfql\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c\" (UID: \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" Oct 30 12:46:57.513195 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:57.513012 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c\" (UID: \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" Oct 30 12:46:57.513195 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:57.513033 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c\" (UID: \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" Oct 30 12:46:57.513932 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:57.513902 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c\" (UID: \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" Oct 30 12:46:57.514095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:57.514010 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c\" (UID: \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" Oct 30 12:46:57.515990 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:57.515964 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c\" (UID: \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" Oct 30 12:46:57.538312 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:57.538277 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vfql\" (UniqueName: \"kubernetes.io/projected/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-kube-api-access-7vfql\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c\" (UID: \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" Oct 30 12:46:57.650847 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:57.650750 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" Oct 30 12:46:57.963566 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:46:57.963534 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44c42d8_7d8c_44d2_8fe2_a2ff432646d2.slice/crio-c0e4bc43eb0e351191def57ba4e9129faa797d6c4a68b6a16575ad2e6d664e25 WatchSource:0}: Error finding container c0e4bc43eb0e351191def57ba4e9129faa797d6c4a68b6a16575ad2e6d664e25: Status 404 returned error can't find the container with id c0e4bc43eb0e351191def57ba4e9129faa797d6c4a68b6a16575ad2e6d664e25 Oct 30 12:46:57.966668 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:57.966640 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c"] Oct 30 12:46:58.064668 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:58.064633 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" event={"ID":"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2","Type":"ContainerStarted","Data":"214cfd288eda54ee2e4b35845cbeb36812ea43d926ea0b5a6dda94cc9d995f35"} Oct 30 12:46:58.064856 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:58.064692 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" event={"ID":"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2","Type":"ContainerStarted","Data":"c0e4bc43eb0e351191def57ba4e9129faa797d6c4a68b6a16575ad2e6d664e25"} Oct 30 12:46:58.064856 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:58.064730 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" Oct 30 12:46:58.133767 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:46:58.133704 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" podStartSLOduration=1.133689305 podStartE2EDuration="1.133689305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:58.130930252 +0000 UTC m=+3169.165068649" watchObservedRunningTime="2026-10-30 12:46:58.133689305 +0000 UTC m=+3169.167827700" Oct 30 12:47:03.717359 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:03.717314 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c"] Oct 30 12:47:03.720916 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:03.720891 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" Oct 30 12:47:08.718258 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:08.718210 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" podUID="a44c42d8-7d8c-44d2-8fe2-a2ff432646d2" containerName="payload-processing" containerID="cri-o://214cfd288eda54ee2e4b35845cbeb36812ea43d926ea0b5a6dda94cc9d995f35" gracePeriod=25 Oct 30 12:47:09.028556 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.028522 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" Oct 30 12:47:09.110567 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.110510 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-kube-api-access-7vfql\" (UniqueName: \"kubernetes.io/projected/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-kube-api-access-7vfql\") pod \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\" (UID: \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\") " Oct 30 12:47:09.110758 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.110627 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-routing-overlay-volume\") pod \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\" (UID: \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\") " Oct 30 12:47:09.110875 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.110859 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-plugins-config-volume\") pod \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\" (UID: \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\") " Oct 30 12:47:09.110958 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.110942 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-provider-credentials\") pod \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\" (UID: \"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2\") " Oct 30 12:47:09.111617 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.111306 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-routing-overlay-volume" pod "a44c42d8-7d8c-44d2-8fe2-a2ff432646d2" (UID: "a44c42d8-7d8c-44d2-8fe2-a2ff432646d2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:47:09.111617 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.111395 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-plugins-config-volume" pod "a44c42d8-7d8c-44d2-8fe2-a2ff432646d2" (UID: "a44c42d8-7d8c-44d2-8fe2-a2ff432646d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:47:09.111617 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.111568 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:47:09.111617 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.111589 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:47:09.113983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.113947 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-provider-credentials" pod "a44c42d8-7d8c-44d2-8fe2-a2ff432646d2" (UID: "a44c42d8-7d8c-44d2-8fe2-a2ff432646d2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:47:09.114905 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.114878 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-kube-api-access-7vfql" pod "a44c42d8-7d8c-44d2-8fe2-a2ff432646d2" (UID: "a44c42d8-7d8c-44d2-8fe2-a2ff432646d2"). InnerVolumeSpecName "kube-api-access-7vfql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:47:09.166398 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.166357 4468 generic.go:356] "Generic (PLEG): container finished" podID="a44c42d8-7d8c-44d2-8fe2-a2ff432646d2" containerID="214cfd288eda54ee2e4b35845cbeb36812ea43d926ea0b5a6dda94cc9d995f35" exitCode=0 Oct 30 12:47:09.166570 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.166416 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" Oct 30 12:47:09.166570 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.166436 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" event={"ID":"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2","Type":"ContainerDied","Data":"214cfd288eda54ee2e4b35845cbeb36812ea43d926ea0b5a6dda94cc9d995f35"} Oct 30 12:47:09.166570 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.166524 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c" event={"ID":"a44c42d8-7d8c-44d2-8fe2-a2ff432646d2","Type":"ContainerDied","Data":"c0e4bc43eb0e351191def57ba4e9129faa797d6c4a68b6a16575ad2e6d664e25"} Oct 30 12:47:09.166570 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.166553 4468 scope.go:122] "RemoveContainer" containerID="214cfd288eda54ee2e4b35845cbeb36812ea43d926ea0b5a6dda94cc9d995f35" Oct 30 12:47:09.185555 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.185531 4468 scope.go:122] "RemoveContainer" containerID="214cfd288eda54ee2e4b35845cbeb36812ea43d926ea0b5a6dda94cc9d995f35" Oct 30 12:47:09.185938 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:47:09.185906 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"214cfd288eda54ee2e4b35845cbeb36812ea43d926ea0b5a6dda94cc9d995f35\": container with ID starting with 214cfd288eda54ee2e4b35845cbeb36812ea43d926ea0b5a6dda94cc9d995f35 not found: ID does not exist" containerID="214cfd288eda54ee2e4b35845cbeb36812ea43d926ea0b5a6dda94cc9d995f35" Oct 30 12:47:09.186046 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.185955 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"214cfd288eda54ee2e4b35845cbeb36812ea43d926ea0b5a6dda94cc9d995f35"} err="failed to get container status \"214cfd288eda54ee2e4b35845cbeb36812ea43d926ea0b5a6dda94cc9d995f35\": rpc error: code = NotFound desc = could not find container \"214cfd288eda54ee2e4b35845cbeb36812ea43d926ea0b5a6dda94cc9d995f35\": container with ID starting with 214cfd288eda54ee2e4b35845cbeb36812ea43d926ea0b5a6dda94cc9d995f35 not found: ID does not exist" Oct 30 12:47:09.213613 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.213578 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:47:09.213613 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.213612 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vfql\" (UniqueName: \"kubernetes.io/projected/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2-kube-api-access-7vfql\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:47:09.227213 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.227178 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c"] Oct 30 12:47:09.232109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:09.232061 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-587dghj5c"] Oct 30 12:47:11.092084 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.092043 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a44c42d8-7d8c-44d2-8fe2-a2ff432646d2" path="/var/lib/kubelet/pods/a44c42d8-7d8c-44d2-8fe2-a2ff432646d2/volumes" Oct 30 12:47:11.714363 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.714324 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5"] Oct 30 12:47:11.715219 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.715197 4468 cpu_manager.go:414] "containerMap: removing container" podUID="a44c42d8-7d8c-44d2-8fe2-a2ff432646d2" containerName="payload-processing" Oct 30 12:47:11.715219 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.715219 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a44c42d8-7d8c-44d2-8fe2-a2ff432646d2" containerName="payload-processing" Oct 30 12:47:11.715430 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.715415 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a44c42d8-7d8c-44d2-8fe2-a2ff432646d2" containerName="payload-processing" Oct 30 12:47:11.720258 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.720237 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" Oct 30 12:47:11.744924 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.744880 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5"] Oct 30 12:47:11.745201 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.745176 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-fddaa3-serving-cert\"" Oct 30 12:47:11.860130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.860056 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrzgk\" (UniqueName: \"kubernetes.io/projected/325f91fe-140a-47b9-8b30-c66476278515-kube-api-access-rrzgk\") pod \"maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5\" (UID: \"325f91fe-140a-47b9-8b30-c66476278515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" Oct 30 12:47:11.860130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.860133 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/325f91fe-140a-47b9-8b30-c66476278515-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5\" (UID: \"325f91fe-140a-47b9-8b30-c66476278515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" Oct 30 12:47:11.860400 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.860223 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/325f91fe-140a-47b9-8b30-c66476278515-metrics-certs\") pod \"maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5\" (UID: \"325f91fe-140a-47b9-8b30-c66476278515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" Oct 30 12:47:11.860400 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.860374 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/325f91fe-140a-47b9-8b30-c66476278515-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5\" (UID: \"325f91fe-140a-47b9-8b30-c66476278515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" Oct 30 12:47:11.961655 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.961621 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/325f91fe-140a-47b9-8b30-c66476278515-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5\" (UID: \"325f91fe-140a-47b9-8b30-c66476278515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" Oct 30 12:47:11.961829 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.961764 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrzgk\" (UniqueName: \"kubernetes.io/projected/325f91fe-140a-47b9-8b30-c66476278515-kube-api-access-rrzgk\") pod \"maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5\" (UID: \"325f91fe-140a-47b9-8b30-c66476278515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" Oct 30 12:47:11.961829 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.961795 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/325f91fe-140a-47b9-8b30-c66476278515-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5\" (UID: \"325f91fe-140a-47b9-8b30-c66476278515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" Oct 30 12:47:11.961829 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.961826 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/325f91fe-140a-47b9-8b30-c66476278515-metrics-certs\") pod \"maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5\" (UID: \"325f91fe-140a-47b9-8b30-c66476278515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" Oct 30 12:47:11.962589 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.962565 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/325f91fe-140a-47b9-8b30-c66476278515-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5\" (UID: \"325f91fe-140a-47b9-8b30-c66476278515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" Oct 30 12:47:11.964289 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.964268 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/325f91fe-140a-47b9-8b30-c66476278515-metrics-certs\") pod \"maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5\" (UID: \"325f91fe-140a-47b9-8b30-c66476278515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" Oct 30 12:47:11.964482 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.964438 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/325f91fe-140a-47b9-8b30-c66476278515-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5\" (UID: \"325f91fe-140a-47b9-8b30-c66476278515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" Oct 30 12:47:11.988260 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:11.988235 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrzgk\" (UniqueName: \"kubernetes.io/projected/325f91fe-140a-47b9-8b30-c66476278515-kube-api-access-rrzgk\") pod \"maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5\" (UID: \"325f91fe-140a-47b9-8b30-c66476278515\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" Oct 30 12:47:12.037166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:12.037127 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" Oct 30 12:47:12.347287 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:47:12.347254 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325f91fe_140a_47b9_8b30_c66476278515.slice/crio-4fc6b1d879430fd9d7a22e0e00267d363194ed33da5e817e186c51a00c797a5d WatchSource:0}: Error finding container 4fc6b1d879430fd9d7a22e0e00267d363194ed33da5e817e186c51a00c797a5d: Status 404 returned error can't find the container with id 4fc6b1d879430fd9d7a22e0e00267d363194ed33da5e817e186c51a00c797a5d Oct 30 12:47:12.348868 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:12.348842 4468 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:47:12.349678 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:12.349658 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5"] Oct 30 12:47:13.214649 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:13.214607 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" event={"ID":"325f91fe-140a-47b9-8b30-c66476278515","Type":"ContainerStarted","Data":"198a4540276fb158ebf1b9b9a1a8e2249924d82ce08dfe5ebc4db0e2f05d7a32"} Oct 30 12:47:13.214854 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:13.214675 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" event={"ID":"325f91fe-140a-47b9-8b30-c66476278515","Type":"ContainerStarted","Data":"4fc6b1d879430fd9d7a22e0e00267d363194ed33da5e817e186c51a00c797a5d"} Oct 30 12:47:13.214854 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:13.214706 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" Oct 30 12:47:13.265764 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:13.265703 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" podStartSLOduration=1.8098449639999998 podStartE2EDuration="2.2656898s" totalImagesPullingTime="455.844836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:13.264876872 +0000 UTC m=+3184.299015267" watchObservedRunningTime="2026-10-30 12:47:13.2656898 +0000 UTC m=+3184.299828195" Oct 30 12:47:19.227381 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:19.227350 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" Oct 30 12:47:23.601302 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:47:23.601202 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:47:29.307903 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:29.307868 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv"] Oct 30 12:47:30.900716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:30.900675 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd"] Oct 30 12:47:34.309008 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:34.308961 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" podUID="da588bf8-ee9a-4103-8da6-45bc7752c3bb" containerName="payload-processing" containerID="cri-o://85cf4177186dde2fbcbad2946d92425f6db0c71cdbd2e515cecc81f451e00a15" gracePeriod=25 Oct 30 12:47:34.340259 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:47:34.340212 4468 logging.go:55] [core] [Channel #10428 SubChannel #10429]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.225:50052", ServerName: "10.134.0.225:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.225:50052: connect: connection refused" Oct 30 12:47:34.608965 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:34.608935 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" Oct 30 12:47:34.694666 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:34.694626 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da588bf8-ee9a-4103-8da6-45bc7752c3bb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/da588bf8-ee9a-4103-8da6-45bc7752c3bb-provider-credentials\") pod \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\" (UID: \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\") " Oct 30 12:47:34.695225 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:34.695194 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da588bf8-ee9a-4103-8da6-45bc7752c3bb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da588bf8-ee9a-4103-8da6-45bc7752c3bb-routing-overlay-volume\") pod \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\" (UID: \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\") " Oct 30 12:47:34.695370 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:34.695295 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da588bf8-ee9a-4103-8da6-45bc7752c3bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da588bf8-ee9a-4103-8da6-45bc7752c3bb-plugins-config-volume\") pod \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\" (UID: \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\") " Oct 30 12:47:34.695370 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:34.695344 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da588bf8-ee9a-4103-8da6-45bc7752c3bb-kube-api-access-hmlh6\" (UniqueName: \"kubernetes.io/projected/da588bf8-ee9a-4103-8da6-45bc7752c3bb-kube-api-access-hmlh6\") pod \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\" (UID: \"da588bf8-ee9a-4103-8da6-45bc7752c3bb\") " Oct 30 12:47:34.695629 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:34.695601 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da588bf8-ee9a-4103-8da6-45bc7752c3bb-routing-overlay-volume" pod "da588bf8-ee9a-4103-8da6-45bc7752c3bb" (UID: "da588bf8-ee9a-4103-8da6-45bc7752c3bb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:47:34.695959 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:34.695938 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da588bf8-ee9a-4103-8da6-45bc7752c3bb-plugins-config-volume" pod "da588bf8-ee9a-4103-8da6-45bc7752c3bb" (UID: "da588bf8-ee9a-4103-8da6-45bc7752c3bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:47:34.696103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:34.695953 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da588bf8-ee9a-4103-8da6-45bc7752c3bb-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:47:34.698118 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:34.698091 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da588bf8-ee9a-4103-8da6-45bc7752c3bb-kube-api-access-hmlh6" pod "da588bf8-ee9a-4103-8da6-45bc7752c3bb" (UID: "da588bf8-ee9a-4103-8da6-45bc7752c3bb"). InnerVolumeSpecName "kube-api-access-hmlh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:47:34.699051 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:34.699020 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da588bf8-ee9a-4103-8da6-45bc7752c3bb-provider-credentials" pod "da588bf8-ee9a-4103-8da6-45bc7752c3bb" (UID: "da588bf8-ee9a-4103-8da6-45bc7752c3bb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:47:34.797884 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:34.797845 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da588bf8-ee9a-4103-8da6-45bc7752c3bb-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:47:34.797884 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:34.797876 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmlh6\" (UniqueName: \"kubernetes.io/projected/da588bf8-ee9a-4103-8da6-45bc7752c3bb-kube-api-access-hmlh6\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:47:34.797884 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:34.797886 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/da588bf8-ee9a-4103-8da6-45bc7752c3bb-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:47:35.340104 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:35.340027 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" podUID="da588bf8-ee9a-4103-8da6-45bc7752c3bb" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.225:50052\" within 1s: context deadline exceeded" Oct 30 12:47:35.419016 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:35.418983 4468 generic.go:356] "Generic (PLEG): container finished" podID="da588bf8-ee9a-4103-8da6-45bc7752c3bb" containerID="85cf4177186dde2fbcbad2946d92425f6db0c71cdbd2e515cecc81f451e00a15" exitCode=0 Oct 30 12:47:35.419255 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:35.419032 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" Oct 30 12:47:35.419255 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:35.419088 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" event={"ID":"da588bf8-ee9a-4103-8da6-45bc7752c3bb","Type":"ContainerDied","Data":"85cf4177186dde2fbcbad2946d92425f6db0c71cdbd2e515cecc81f451e00a15"} Oct 30 12:47:35.419255 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:35.419155 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv" event={"ID":"da588bf8-ee9a-4103-8da6-45bc7752c3bb","Type":"ContainerDied","Data":"bc830a31ddd4b3e68a6d3ec468248c2f7bb1c0e93da72891954811b943a48863"} Oct 30 12:47:35.419255 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:35.419186 4468 scope.go:122] "RemoveContainer" containerID="85cf4177186dde2fbcbad2946d92425f6db0c71cdbd2e515cecc81f451e00a15" Oct 30 12:47:35.429669 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:35.429647 4468 scope.go:122] "RemoveContainer" containerID="85cf4177186dde2fbcbad2946d92425f6db0c71cdbd2e515cecc81f451e00a15" Oct 30 12:47:35.430029 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:47:35.429997 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85cf4177186dde2fbcbad2946d92425f6db0c71cdbd2e515cecc81f451e00a15\": container with ID starting with 85cf4177186dde2fbcbad2946d92425f6db0c71cdbd2e515cecc81f451e00a15 not found: ID does not exist" containerID="85cf4177186dde2fbcbad2946d92425f6db0c71cdbd2e515cecc81f451e00a15" Oct 30 12:47:35.430130 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:35.430038 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85cf4177186dde2fbcbad2946d92425f6db0c71cdbd2e515cecc81f451e00a15"} err="failed to get container status \"85cf4177186dde2fbcbad2946d92425f6db0c71cdbd2e515cecc81f451e00a15\": rpc error: code = NotFound desc = could not find container \"85cf4177186dde2fbcbad2946d92425f6db0c71cdbd2e515cecc81f451e00a15\": container with ID starting with 85cf4177186dde2fbcbad2946d92425f6db0c71cdbd2e515cecc81f451e00a15 not found: ID does not exist" Oct 30 12:47:35.477821 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:35.477785 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv"] Oct 30 12:47:35.491576 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:35.491541 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-677cxnwkv"] Oct 30 12:47:35.901637 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:35.901528 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" podUID="748d03d3-bf28-4060-8821-8b558864dafb" containerName="payload-processing" containerID="cri-o://4eba19738e5d7c295e5a0ff7c4d1481086c3aa0c4c48dc4ba4ac80ddeb472ae4" gracePeriod=25 Oct 30 12:47:36.193547 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.193514 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" Oct 30 12:47:36.318233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.318202 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748d03d3-bf28-4060-8821-8b558864dafb-kube-api-access-b92d9\" (UniqueName: \"kubernetes.io/projected/748d03d3-bf28-4060-8821-8b558864dafb-kube-api-access-b92d9\") pod \"748d03d3-bf28-4060-8821-8b558864dafb\" (UID: \"748d03d3-bf28-4060-8821-8b558864dafb\") " Oct 30 12:47:36.318416 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.318355 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748d03d3-bf28-4060-8821-8b558864dafb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/748d03d3-bf28-4060-8821-8b558864dafb-provider-credentials\") pod \"748d03d3-bf28-4060-8821-8b558864dafb\" (UID: \"748d03d3-bf28-4060-8821-8b558864dafb\") " Oct 30 12:47:36.318492 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.318418 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/748d03d3-bf28-4060-8821-8b558864dafb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/748d03d3-bf28-4060-8821-8b558864dafb-routing-overlay-volume\") pod \"748d03d3-bf28-4060-8821-8b558864dafb\" (UID: \"748d03d3-bf28-4060-8821-8b558864dafb\") " Oct 30 12:47:36.318492 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.318442 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/748d03d3-bf28-4060-8821-8b558864dafb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/748d03d3-bf28-4060-8821-8b558864dafb-plugins-config-volume\") pod \"748d03d3-bf28-4060-8821-8b558864dafb\" (UID: \"748d03d3-bf28-4060-8821-8b558864dafb\") " Oct 30 12:47:36.318826 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.318790 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/748d03d3-bf28-4060-8821-8b558864dafb-routing-overlay-volume" pod "748d03d3-bf28-4060-8821-8b558864dafb" (UID: "748d03d3-bf28-4060-8821-8b558864dafb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:47:36.318950 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.318919 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/748d03d3-bf28-4060-8821-8b558864dafb-plugins-config-volume" pod "748d03d3-bf28-4060-8821-8b558864dafb" (UID: "748d03d3-bf28-4060-8821-8b558864dafb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:47:36.319091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.319048 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/748d03d3-bf28-4060-8821-8b558864dafb-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:47:36.319164 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.319095 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/748d03d3-bf28-4060-8821-8b558864dafb-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:47:36.320668 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.320645 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748d03d3-bf28-4060-8821-8b558864dafb-kube-api-access-b92d9" pod "748d03d3-bf28-4060-8821-8b558864dafb" (UID: "748d03d3-bf28-4060-8821-8b558864dafb"). InnerVolumeSpecName "kube-api-access-b92d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:47:36.320792 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.320674 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748d03d3-bf28-4060-8821-8b558864dafb-provider-credentials" pod "748d03d3-bf28-4060-8821-8b558864dafb" (UID: "748d03d3-bf28-4060-8821-8b558864dafb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:47:36.421174 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.421085 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b92d9\" (UniqueName: \"kubernetes.io/projected/748d03d3-bf28-4060-8821-8b558864dafb-kube-api-access-b92d9\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:47:36.421174 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.421123 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/748d03d3-bf28-4060-8821-8b558864dafb-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:47:36.429189 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.429149 4468 generic.go:356] "Generic (PLEG): container finished" podID="748d03d3-bf28-4060-8821-8b558864dafb" containerID="4eba19738e5d7c295e5a0ff7c4d1481086c3aa0c4c48dc4ba4ac80ddeb472ae4" exitCode=0 Oct 30 12:47:36.429387 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.429204 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" Oct 30 12:47:36.429387 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.429233 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" event={"ID":"748d03d3-bf28-4060-8821-8b558864dafb","Type":"ContainerDied","Data":"4eba19738e5d7c295e5a0ff7c4d1481086c3aa0c4c48dc4ba4ac80ddeb472ae4"} Oct 30 12:47:36.429387 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.429327 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd" event={"ID":"748d03d3-bf28-4060-8821-8b558864dafb","Type":"ContainerDied","Data":"bb2dbd0dae513ecc825dbde954bf6fb24fcd77f877af64fcaf749049d286264a"} Oct 30 12:47:36.429387 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.429359 4468 scope.go:122] "RemoveContainer" containerID="4eba19738e5d7c295e5a0ff7c4d1481086c3aa0c4c48dc4ba4ac80ddeb472ae4" Oct 30 12:47:36.441018 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.440997 4468 scope.go:122] "RemoveContainer" containerID="4eba19738e5d7c295e5a0ff7c4d1481086c3aa0c4c48dc4ba4ac80ddeb472ae4" Oct 30 12:47:36.441405 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:47:36.441384 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eba19738e5d7c295e5a0ff7c4d1481086c3aa0c4c48dc4ba4ac80ddeb472ae4\": container with ID starting with 4eba19738e5d7c295e5a0ff7c4d1481086c3aa0c4c48dc4ba4ac80ddeb472ae4 not found: ID does not exist" containerID="4eba19738e5d7c295e5a0ff7c4d1481086c3aa0c4c48dc4ba4ac80ddeb472ae4" Oct 30 12:47:36.441466 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.441417 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eba19738e5d7c295e5a0ff7c4d1481086c3aa0c4c48dc4ba4ac80ddeb472ae4"} err="failed to get container status \"4eba19738e5d7c295e5a0ff7c4d1481086c3aa0c4c48dc4ba4ac80ddeb472ae4\": rpc error: code = NotFound desc = could not find container \"4eba19738e5d7c295e5a0ff7c4d1481086c3aa0c4c48dc4ba4ac80ddeb472ae4\": container with ID starting with 4eba19738e5d7c295e5a0ff7c4d1481086c3aa0c4c48dc4ba4ac80ddeb472ae4 not found: ID does not exist" Oct 30 12:47:36.487914 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.487880 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd"] Oct 30 12:47:36.499707 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:36.499656 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-788562khd"] Oct 30 12:47:37.092030 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:37.091996 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="748d03d3-bf28-4060-8821-8b558864dafb" path="/var/lib/kubelet/pods/748d03d3-bf28-4060-8821-8b558864dafb/volumes" Oct 30 12:47:37.092497 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:37.092482 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da588bf8-ee9a-4103-8da6-45bc7752c3bb" path="/var/lib/kubelet/pods/da588bf8-ee9a-4103-8da6-45bc7752c3bb/volumes" Oct 30 12:47:41.297699 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:41.297658 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2"] Oct 30 12:47:45.142650 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.142609 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj"] Oct 30 12:47:45.143594 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.143573 4468 cpu_manager.go:414] "containerMap: removing container" podUID="da588bf8-ee9a-4103-8da6-45bc7752c3bb" containerName="payload-processing" Oct 30 12:47:45.143675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.143595 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da588bf8-ee9a-4103-8da6-45bc7752c3bb" containerName="payload-processing" Oct 30 12:47:45.143675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.143638 4468 cpu_manager.go:414] "containerMap: removing container" podUID="748d03d3-bf28-4060-8821-8b558864dafb" containerName="payload-processing" Oct 30 12:47:45.143675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.143646 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="748d03d3-bf28-4060-8821-8b558864dafb" containerName="payload-processing" Oct 30 12:47:45.143787 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.143777 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="748d03d3-bf28-4060-8821-8b558864dafb" containerName="payload-processing" Oct 30 12:47:45.143829 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.143795 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="da588bf8-ee9a-4103-8da6-45bc7752c3bb" containerName="payload-processing" Oct 30 12:47:45.147647 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.147628 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" Oct 30 12:47:45.167831 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.167791 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj"] Oct 30 12:47:45.174710 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.174676 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"agc-cred-5791f6dab64a\"" Oct 30 12:47:45.174895 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.174747 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"payload-processing-external-model-e2e--3f99fe6c-dockercfg-npqld\"" Oct 30 12:47:45.258116 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.258049 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2cc7ee8-0f74-45aa-944f-71191679478a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj\" (UID: \"a2cc7ee8-0f74-45aa-944f-71191679478a\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" Oct 30 12:47:45.258389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.258368 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t867v\" (UniqueName: \"kubernetes.io/projected/a2cc7ee8-0f74-45aa-944f-71191679478a-kube-api-access-t867v\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj\" (UID: \"a2cc7ee8-0f74-45aa-944f-71191679478a\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" Oct 30 12:47:45.258679 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.258659 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2cc7ee8-0f74-45aa-944f-71191679478a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj\" (UID: \"a2cc7ee8-0f74-45aa-944f-71191679478a\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" Oct 30 12:47:45.258807 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.258792 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2cc7ee8-0f74-45aa-944f-71191679478a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj\" (UID: \"a2cc7ee8-0f74-45aa-944f-71191679478a\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" Oct 30 12:47:45.361505 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.361456 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2cc7ee8-0f74-45aa-944f-71191679478a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj\" (UID: \"a2cc7ee8-0f74-45aa-944f-71191679478a\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" Oct 30 12:47:45.361775 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.361755 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2cc7ee8-0f74-45aa-944f-71191679478a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj\" (UID: \"a2cc7ee8-0f74-45aa-944f-71191679478a\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" Oct 30 12:47:45.362097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.362059 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2cc7ee8-0f74-45aa-944f-71191679478a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj\" (UID: \"a2cc7ee8-0f74-45aa-944f-71191679478a\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" Oct 30 12:47:45.362248 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.362232 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t867v\" (UniqueName: \"kubernetes.io/projected/a2cc7ee8-0f74-45aa-944f-71191679478a-kube-api-access-t867v\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj\" (UID: \"a2cc7ee8-0f74-45aa-944f-71191679478a\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" Oct 30 12:47:45.364345 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.364002 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2cc7ee8-0f74-45aa-944f-71191679478a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj\" (UID: \"a2cc7ee8-0f74-45aa-944f-71191679478a\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" Oct 30 12:47:45.367955 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.367929 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2cc7ee8-0f74-45aa-944f-71191679478a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj\" (UID: \"a2cc7ee8-0f74-45aa-944f-71191679478a\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" Oct 30 12:47:45.368324 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.368300 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2cc7ee8-0f74-45aa-944f-71191679478a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj\" (UID: \"a2cc7ee8-0f74-45aa-944f-71191679478a\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" Oct 30 12:47:45.399385 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.399252 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t867v\" (UniqueName: \"kubernetes.io/projected/a2cc7ee8-0f74-45aa-944f-71191679478a-kube-api-access-t867v\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj\" (UID: \"a2cc7ee8-0f74-45aa-944f-71191679478a\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" Oct 30 12:47:45.463663 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.463619 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" Oct 30 12:47:45.764135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.764097 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2"] Oct 30 12:47:45.769734 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.769708 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" Oct 30 12:47:45.796357 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.796323 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"agc-cred-1adfe4d5b69e\"" Oct 30 12:47:45.801266 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.801206 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2"] Oct 30 12:47:45.836947 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.836899 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj"] Oct 30 12:47:45.848647 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:47:45.848611 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2cc7ee8_0f74_45aa_944f_71191679478a.slice/crio-6cbae37d5eb31c8ca1d1d27a60e4f4fc25f30709c186193941c398307087d68c WatchSource:0}: Error finding container 6cbae37d5eb31c8ca1d1d27a60e4f4fc25f30709c186193941c398307087d68c: Status 404 returned error can't find the container with id 6cbae37d5eb31c8ca1d1d27a60e4f4fc25f30709c186193941c398307087d68c Oct 30 12:47:45.870651 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.870615 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9391811-2e96-4b3d-879a-c631c4896ed8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2\" (UID: \"e9391811-2e96-4b3d-879a-c631c4896ed8\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" Oct 30 12:47:45.870985 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.870955 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9391811-2e96-4b3d-879a-c631c4896ed8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2\" (UID: \"e9391811-2e96-4b3d-879a-c631c4896ed8\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" Oct 30 12:47:45.871154 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.871023 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9391811-2e96-4b3d-879a-c631c4896ed8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2\" (UID: \"e9391811-2e96-4b3d-879a-c631c4896ed8\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" Oct 30 12:47:45.871154 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.871112 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjfcs\" (UniqueName: \"kubernetes.io/projected/e9391811-2e96-4b3d-879a-c631c4896ed8-kube-api-access-sjfcs\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2\" (UID: \"e9391811-2e96-4b3d-879a-c631c4896ed8\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" Oct 30 12:47:45.972261 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.972221 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9391811-2e96-4b3d-879a-c631c4896ed8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2\" (UID: \"e9391811-2e96-4b3d-879a-c631c4896ed8\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" Oct 30 12:47:45.972470 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.972450 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9391811-2e96-4b3d-879a-c631c4896ed8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2\" (UID: \"e9391811-2e96-4b3d-879a-c631c4896ed8\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" Oct 30 12:47:45.972550 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.972512 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9391811-2e96-4b3d-879a-c631c4896ed8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2\" (UID: \"e9391811-2e96-4b3d-879a-c631c4896ed8\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" Oct 30 12:47:45.972613 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.972559 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjfcs\" (UniqueName: \"kubernetes.io/projected/e9391811-2e96-4b3d-879a-c631c4896ed8-kube-api-access-sjfcs\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2\" (UID: \"e9391811-2e96-4b3d-879a-c631c4896ed8\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" Oct 30 12:47:45.973731 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.973380 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9391811-2e96-4b3d-879a-c631c4896ed8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2\" (UID: \"e9391811-2e96-4b3d-879a-c631c4896ed8\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" Oct 30 12:47:45.973731 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.973553 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9391811-2e96-4b3d-879a-c631c4896ed8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2\" (UID: \"e9391811-2e96-4b3d-879a-c631c4896ed8\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" Oct 30 12:47:45.978659 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:45.978632 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9391811-2e96-4b3d-879a-c631c4896ed8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2\" (UID: \"e9391811-2e96-4b3d-879a-c631c4896ed8\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" Oct 30 12:47:46.006034 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.005994 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjfcs\" (UniqueName: \"kubernetes.io/projected/e9391811-2e96-4b3d-879a-c631c4896ed8-kube-api-access-sjfcs\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2\" (UID: \"e9391811-2e96-4b3d-879a-c631c4896ed8\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" Oct 30 12:47:46.093167 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.092690 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" Oct 30 12:47:46.298598 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.298549 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" podUID="ce74d0e0-2e5a-4213-a460-7430ca8f3cb7" containerName="payload-processing" containerID="cri-o://1b9818f202e732518f56b73c09de93cf8d5f355a0af0c7125f2ef31e2e8feac8" gracePeriod=25 Oct 30 12:47:46.485734 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:47:46.485694 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9391811_2e96_4b3d_879a_c631c4896ed8.slice/crio-eb754ef90f20cd83d371303af6a7bf9aec70278090e3cb75987ff703f38a1d14 WatchSource:0}: Error finding container eb754ef90f20cd83d371303af6a7bf9aec70278090e3cb75987ff703f38a1d14: Status 404 returned error can't find the container with id eb754ef90f20cd83d371303af6a7bf9aec70278090e3cb75987ff703f38a1d14 Oct 30 12:47:46.487472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.487440 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2"] Oct 30 12:47:46.557667 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.557619 4468 generic.go:356] "Generic (PLEG): container finished" podID="ce74d0e0-2e5a-4213-a460-7430ca8f3cb7" containerID="1b9818f202e732518f56b73c09de93cf8d5f355a0af0c7125f2ef31e2e8feac8" exitCode=0 Oct 30 12:47:46.557866 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.557746 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" event={"ID":"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7","Type":"ContainerDied","Data":"1b9818f202e732518f56b73c09de93cf8d5f355a0af0c7125f2ef31e2e8feac8"} Oct 30 12:47:46.559385 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.559350 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" event={"ID":"e9391811-2e96-4b3d-879a-c631c4896ed8","Type":"ContainerStarted","Data":"eb754ef90f20cd83d371303af6a7bf9aec70278090e3cb75987ff703f38a1d14"} Oct 30 12:47:46.563426 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.563397 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" event={"ID":"a2cc7ee8-0f74-45aa-944f-71191679478a","Type":"ContainerStarted","Data":"65ff1a124a77ee271f31895a287670d92b7f4378fe925594b1b1dde1cba7a08d"} Oct 30 12:47:46.563548 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.563451 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" event={"ID":"a2cc7ee8-0f74-45aa-944f-71191679478a","Type":"ContainerStarted","Data":"6cbae37d5eb31c8ca1d1d27a60e4f4fc25f30709c186193941c398307087d68c"} Oct 30 12:47:46.564235 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.564214 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" Oct 30 12:47:46.622350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.621121 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" podStartSLOduration=1.621099434 podStartE2EDuration="1.621099434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:46.610692537 +0000 UTC m=+3217.644830946" watchObservedRunningTime="2026-10-30 12:47:46.621099434 +0000 UTC m=+3217.655237831" Oct 30 12:47:46.667894 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.667858 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" Oct 30 12:47:46.685718 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.685657 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-provider-credentials\") pod \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\" (UID: \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\") " Oct 30 12:47:46.685899 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.685792 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-kube-api-access-6xpgw\" (UniqueName: \"kubernetes.io/projected/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-kube-api-access-6xpgw\") pod \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\" (UID: \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\") " Oct 30 12:47:46.685899 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.685869 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-plugins-config-volume\") pod \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\" (UID: \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\") " Oct 30 12:47:46.686047 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.685934 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-routing-overlay-volume\") pod \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\" (UID: \"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7\") " Oct 30 12:47:46.686520 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.686469 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-routing-overlay-volume" pod "ce74d0e0-2e5a-4213-a460-7430ca8f3cb7" (UID: "ce74d0e0-2e5a-4213-a460-7430ca8f3cb7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:47:46.686639 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.686598 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-plugins-config-volume" pod "ce74d0e0-2e5a-4213-a460-7430ca8f3cb7" (UID: "ce74d0e0-2e5a-4213-a460-7430ca8f3cb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:47:46.688347 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.688321 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-kube-api-access-6xpgw" pod "ce74d0e0-2e5a-4213-a460-7430ca8f3cb7" (UID: "ce74d0e0-2e5a-4213-a460-7430ca8f3cb7"). InnerVolumeSpecName "kube-api-access-6xpgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:47:46.688432 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.688366 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-provider-credentials" pod "ce74d0e0-2e5a-4213-a460-7430ca8f3cb7" (UID: "ce74d0e0-2e5a-4213-a460-7430ca8f3cb7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:47:46.787584 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.787537 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xpgw\" (UniqueName: \"kubernetes.io/projected/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-kube-api-access-6xpgw\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:47:46.787584 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.787582 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:47:46.787584 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.787594 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:47:46.787850 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:46.787602 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:47:47.572284 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:47.572181 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" event={"ID":"e9391811-2e96-4b3d-879a-c631c4896ed8","Type":"ContainerStarted","Data":"0589e22b0e8e15d80e07d663722dea7080ce06a392d7f711a6238636b6ca67c0"} Oct 30 12:47:47.572713 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:47.572283 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" Oct 30 12:47:47.573980 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:47.573955 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" event={"ID":"ce74d0e0-2e5a-4213-a460-7430ca8f3cb7","Type":"ContainerDied","Data":"6021fb4ed0915a3b855a30cccc3879526fd03755748911f0bc6bb5aeddc16dee"} Oct 30 12:47:47.574137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:47.573990 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2" Oct 30 12:47:47.574137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:47.574010 4468 scope.go:122] "RemoveContainer" containerID="1b9818f202e732518f56b73c09de93cf8d5f355a0af0c7125f2ef31e2e8feac8" Oct 30 12:47:47.620875 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:47.620813 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" podStartSLOduration=2.62079882 podStartE2EDuration="2.62079882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:47.616189155 +0000 UTC m=+3218.650327551" watchObservedRunningTime="2026-10-30 12:47:47.62079882 +0000 UTC m=+3218.654937215" Oct 30 12:47:47.658757 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:47.658723 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2"] Oct 30 12:47:47.664117 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:47.664086 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-58dccwlb2"] Oct 30 12:47:49.093979 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:49.093945 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce74d0e0-2e5a-4213-a460-7430ca8f3cb7" path="/var/lib/kubelet/pods/ce74d0e0-2e5a-4213-a460-7430ca8f3cb7/volumes" Oct 30 12:47:53.601993 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:47:53.601957 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:47:57.505864 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.505822 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb"] Oct 30 12:47:57.507818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.506712 4468 cpu_manager.go:414] "containerMap: removing container" podUID="ce74d0e0-2e5a-4213-a460-7430ca8f3cb7" containerName="payload-processing" Oct 30 12:47:57.507818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.506730 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce74d0e0-2e5a-4213-a460-7430ca8f3cb7" containerName="payload-processing" Oct 30 12:47:57.507818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.506874 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce74d0e0-2e5a-4213-a460-7430ca8f3cb7" containerName="payload-processing" Oct 30 12:47:57.510782 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.510760 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" Oct 30 12:47:57.542434 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.542397 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"agc-cred-d510e13eefb7\"" Oct 30 12:47:57.548845 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.548810 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb"] Oct 30 12:47:57.584950 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.584893 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7495z\" (UniqueName: \"kubernetes.io/projected/7b12aac3-8f71-487d-aba8-9def1e7439a3-kube-api-access-7495z\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb\" (UID: \"7b12aac3-8f71-487d-aba8-9def1e7439a3\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" Oct 30 12:47:57.585225 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.585129 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b12aac3-8f71-487d-aba8-9def1e7439a3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb\" (UID: \"7b12aac3-8f71-487d-aba8-9def1e7439a3\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" Oct 30 12:47:57.585312 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.585232 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b12aac3-8f71-487d-aba8-9def1e7439a3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb\" (UID: \"7b12aac3-8f71-487d-aba8-9def1e7439a3\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" Oct 30 12:47:57.585312 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.585283 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b12aac3-8f71-487d-aba8-9def1e7439a3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb\" (UID: \"7b12aac3-8f71-487d-aba8-9def1e7439a3\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" Oct 30 12:47:57.638362 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.638318 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" Oct 30 12:47:57.687575 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.687523 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7495z\" (UniqueName: \"kubernetes.io/projected/7b12aac3-8f71-487d-aba8-9def1e7439a3-kube-api-access-7495z\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb\" (UID: \"7b12aac3-8f71-487d-aba8-9def1e7439a3\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" Oct 30 12:47:57.687818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.687783 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b12aac3-8f71-487d-aba8-9def1e7439a3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb\" (UID: \"7b12aac3-8f71-487d-aba8-9def1e7439a3\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" Oct 30 12:47:57.687910 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.687891 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b12aac3-8f71-487d-aba8-9def1e7439a3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb\" (UID: \"7b12aac3-8f71-487d-aba8-9def1e7439a3\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" Oct 30 12:47:57.687992 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.687952 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b12aac3-8f71-487d-aba8-9def1e7439a3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb\" (UID: \"7b12aac3-8f71-487d-aba8-9def1e7439a3\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" Oct 30 12:47:57.689001 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.688968 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b12aac3-8f71-487d-aba8-9def1e7439a3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb\" (UID: \"7b12aac3-8f71-487d-aba8-9def1e7439a3\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" Oct 30 12:47:57.689283 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.689245 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b12aac3-8f71-487d-aba8-9def1e7439a3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb\" (UID: \"7b12aac3-8f71-487d-aba8-9def1e7439a3\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" Oct 30 12:47:57.692922 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.692885 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b12aac3-8f71-487d-aba8-9def1e7439a3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb\" (UID: \"7b12aac3-8f71-487d-aba8-9def1e7439a3\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" Oct 30 12:47:57.713962 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.713884 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7495z\" (UniqueName: \"kubernetes.io/projected/7b12aac3-8f71-487d-aba8-9def1e7439a3-kube-api-access-7495z\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb\" (UID: \"7b12aac3-8f71-487d-aba8-9def1e7439a3\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" Oct 30 12:47:57.826182 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:57.826098 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" Oct 30 12:47:58.148308 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:47:58.148278 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b12aac3_8f71_487d_aba8_9def1e7439a3.slice/crio-eef7aae6a4b227aa7f3e9528f6620c1f75149a637d621050752e35b63489d261 WatchSource:0}: Error finding container eef7aae6a4b227aa7f3e9528f6620c1f75149a637d621050752e35b63489d261: Status 404 returned error can't find the container with id eef7aae6a4b227aa7f3e9528f6620c1f75149a637d621050752e35b63489d261 Oct 30 12:47:58.151816 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.151779 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb"] Oct 30 12:47:58.329596 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.329553 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb"] Oct 30 12:47:58.374440 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.374403 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd"] Oct 30 12:47:58.379366 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.379340 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" Oct 30 12:47:58.395470 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.395392 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"agc-cred-b56375d39e66\"" Oct 30 12:47:58.397590 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.397537 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef130a03-b77d-4f73-a975-33f09df99896-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd\" (UID: \"ef130a03-b77d-4f73-a975-33f09df99896\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" Oct 30 12:47:58.397932 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.397902 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8lpf\" (UniqueName: \"kubernetes.io/projected/ef130a03-b77d-4f73-a975-33f09df99896-kube-api-access-n8lpf\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd\" (UID: \"ef130a03-b77d-4f73-a975-33f09df99896\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" Oct 30 12:47:58.398145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.398117 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef130a03-b77d-4f73-a975-33f09df99896-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd\" (UID: \"ef130a03-b77d-4f73-a975-33f09df99896\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" Oct 30 12:47:58.398225 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.398173 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef130a03-b77d-4f73-a975-33f09df99896-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd\" (UID: \"ef130a03-b77d-4f73-a975-33f09df99896\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" Oct 30 12:47:58.398866 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.398844 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd"] Oct 30 12:47:58.499418 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.499380 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef130a03-b77d-4f73-a975-33f09df99896-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd\" (UID: \"ef130a03-b77d-4f73-a975-33f09df99896\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" Oct 30 12:47:58.499598 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.499436 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef130a03-b77d-4f73-a975-33f09df99896-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd\" (UID: \"ef130a03-b77d-4f73-a975-33f09df99896\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" Oct 30 12:47:58.499689 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.499668 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef130a03-b77d-4f73-a975-33f09df99896-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd\" (UID: \"ef130a03-b77d-4f73-a975-33f09df99896\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" Oct 30 12:47:58.499767 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.499751 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8lpf\" (UniqueName: \"kubernetes.io/projected/ef130a03-b77d-4f73-a975-33f09df99896-kube-api-access-n8lpf\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd\" (UID: \"ef130a03-b77d-4f73-a975-33f09df99896\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" Oct 30 12:47:58.500483 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.500456 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef130a03-b77d-4f73-a975-33f09df99896-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd\" (UID: \"ef130a03-b77d-4f73-a975-33f09df99896\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" Oct 30 12:47:58.500609 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.500456 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef130a03-b77d-4f73-a975-33f09df99896-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd\" (UID: \"ef130a03-b77d-4f73-a975-33f09df99896\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" Oct 30 12:47:58.502290 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.502266 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef130a03-b77d-4f73-a975-33f09df99896-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd\" (UID: \"ef130a03-b77d-4f73-a975-33f09df99896\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" Oct 30 12:47:58.525104 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.525040 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8lpf\" (UniqueName: \"kubernetes.io/projected/ef130a03-b77d-4f73-a975-33f09df99896-kube-api-access-n8lpf\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd\" (UID: \"ef130a03-b77d-4f73-a975-33f09df99896\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" Oct 30 12:47:58.585530 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.585502 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" Oct 30 12:47:58.671514 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.671473 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj"] Oct 30 12:47:58.693669 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.693622 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" event={"ID":"7b12aac3-8f71-487d-aba8-9def1e7439a3","Type":"ContainerStarted","Data":"06c7307c969337e0264cc06c586b0b6b3ed06142d866b71674aee177211ccca8"} Oct 30 12:47:58.693876 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.693721 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" event={"ID":"7b12aac3-8f71-487d-aba8-9def1e7439a3","Type":"ContainerStarted","Data":"eef7aae6a4b227aa7f3e9528f6620c1f75149a637d621050752e35b63489d261"} Oct 30 12:47:58.693876 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.693756 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" Oct 30 12:47:58.695788 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.695761 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" Oct 30 12:47:58.769508 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:58.769447 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" podStartSLOduration=1.769428642 podStartE2EDuration="1.769428642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:58.760589868 +0000 UTC m=+3229.794728263" watchObservedRunningTime="2026-10-30 12:47:58.769428642 +0000 UTC m=+3229.803567043" Oct 30 12:47:59.025721 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:59.025595 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd"] Oct 30 12:47:59.702539 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:59.702502 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" event={"ID":"ef130a03-b77d-4f73-a975-33f09df99896","Type":"ContainerStarted","Data":"10e5ab60e0a09298387f37853014477e4edc4ce5d14edef3352cf2a32c37db2c"} Oct 30 12:47:59.702929 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:59.702563 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" event={"ID":"ef130a03-b77d-4f73-a975-33f09df99896","Type":"ContainerStarted","Data":"b6eb453b354ffe27a5281417ad75ed184e2d0b0e1bc003db16fad2ca86e4fc58"} Oct 30 12:47:59.702929 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:59.702752 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" Oct 30 12:47:59.801871 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:47:59.801815 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" podStartSLOduration=1.8017999260000002 podStartE2EDuration="1.801799926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:59.80073931 +0000 UTC m=+3230.834877705" watchObservedRunningTime="2026-10-30 12:47:59.801799926 +0000 UTC m=+3230.835938321" Oct 30 12:48:03.671858 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:03.671810 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" podUID="a2cc7ee8-0f74-45aa-944f-71191679478a" containerName="payload-processing" containerID="cri-o://65ff1a124a77ee271f31895a287670d92b7f4378fe925594b1b1dde1cba7a08d" gracePeriod=25 Oct 30 12:48:03.981058 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:03.981030 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" Oct 30 12:48:04.020412 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.020366 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2cc7ee8-0f74-45aa-944f-71191679478a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2cc7ee8-0f74-45aa-944f-71191679478a-provider-credentials\") pod \"a2cc7ee8-0f74-45aa-944f-71191679478a\" (UID: \"a2cc7ee8-0f74-45aa-944f-71191679478a\") " Oct 30 12:48:04.020635 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.020467 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2cc7ee8-0f74-45aa-944f-71191679478a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2cc7ee8-0f74-45aa-944f-71191679478a-routing-overlay-volume\") pod \"a2cc7ee8-0f74-45aa-944f-71191679478a\" (UID: \"a2cc7ee8-0f74-45aa-944f-71191679478a\") " Oct 30 12:48:04.020635 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.020622 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2cc7ee8-0f74-45aa-944f-71191679478a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2cc7ee8-0f74-45aa-944f-71191679478a-plugins-config-volume\") pod \"a2cc7ee8-0f74-45aa-944f-71191679478a\" (UID: \"a2cc7ee8-0f74-45aa-944f-71191679478a\") " Oct 30 12:48:04.021164 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.020744 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2cc7ee8-0f74-45aa-944f-71191679478a-kube-api-access-t867v\" (UniqueName: \"kubernetes.io/projected/a2cc7ee8-0f74-45aa-944f-71191679478a-kube-api-access-t867v\") pod \"a2cc7ee8-0f74-45aa-944f-71191679478a\" (UID: \"a2cc7ee8-0f74-45aa-944f-71191679478a\") " Oct 30 12:48:04.021164 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.021026 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2cc7ee8-0f74-45aa-944f-71191679478a-routing-overlay-volume" pod "a2cc7ee8-0f74-45aa-944f-71191679478a" (UID: "a2cc7ee8-0f74-45aa-944f-71191679478a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:48:04.021302 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.021206 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2cc7ee8-0f74-45aa-944f-71191679478a-plugins-config-volume" pod "a2cc7ee8-0f74-45aa-944f-71191679478a" (UID: "a2cc7ee8-0f74-45aa-944f-71191679478a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:48:04.021533 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.021512 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2cc7ee8-0f74-45aa-944f-71191679478a-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:48:04.021638 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.021537 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2cc7ee8-0f74-45aa-944f-71191679478a-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:48:04.022999 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.022975 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2cc7ee8-0f74-45aa-944f-71191679478a-provider-credentials" pod "a2cc7ee8-0f74-45aa-944f-71191679478a" (UID: "a2cc7ee8-0f74-45aa-944f-71191679478a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:48:04.023284 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.023267 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2cc7ee8-0f74-45aa-944f-71191679478a-kube-api-access-t867v" pod "a2cc7ee8-0f74-45aa-944f-71191679478a" (UID: "a2cc7ee8-0f74-45aa-944f-71191679478a"). InnerVolumeSpecName "kube-api-access-t867v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:48:04.123356 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.123323 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t867v\" (UniqueName: \"kubernetes.io/projected/a2cc7ee8-0f74-45aa-944f-71191679478a-kube-api-access-t867v\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:48:04.123356 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.123354 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2cc7ee8-0f74-45aa-944f-71191679478a-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:48:04.703306 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.703258 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" podUID="7b12aac3-8f71-487d-aba8-9def1e7439a3" containerName="payload-processing" containerID="cri-o://06c7307c969337e0264cc06c586b0b6b3ed06142d866b71674aee177211ccca8" gracePeriod=25 Oct 30 12:48:04.829769 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.829727 4468 generic.go:356] "Generic (PLEG): container finished" podID="a2cc7ee8-0f74-45aa-944f-71191679478a" containerID="65ff1a124a77ee271f31895a287670d92b7f4378fe925594b1b1dde1cba7a08d" exitCode=0 Oct 30 12:48:04.829999 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.829979 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" event={"ID":"a2cc7ee8-0f74-45aa-944f-71191679478a","Type":"ContainerDied","Data":"65ff1a124a77ee271f31895a287670d92b7f4378fe925594b1b1dde1cba7a08d"} Oct 30 12:48:04.830192 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.830037 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" event={"ID":"a2cc7ee8-0f74-45aa-944f-71191679478a","Type":"ContainerDied","Data":"6cbae37d5eb31c8ca1d1d27a60e4f4fc25f30709c186193941c398307087d68c"} Oct 30 12:48:04.830192 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.830085 4468 scope.go:122] "RemoveContainer" containerID="65ff1a124a77ee271f31895a287670d92b7f4378fe925594b1b1dde1cba7a08d" Oct 30 12:48:04.830326 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.830255 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj" Oct 30 12:48:04.856563 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.856529 4468 scope.go:122] "RemoveContainer" containerID="65ff1a124a77ee271f31895a287670d92b7f4378fe925594b1b1dde1cba7a08d" Oct 30 12:48:04.857296 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:48:04.857259 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ff1a124a77ee271f31895a287670d92b7f4378fe925594b1b1dde1cba7a08d\": container with ID starting with 65ff1a124a77ee271f31895a287670d92b7f4378fe925594b1b1dde1cba7a08d not found: ID does not exist" containerID="65ff1a124a77ee271f31895a287670d92b7f4378fe925594b1b1dde1cba7a08d" Oct 30 12:48:04.857424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.857303 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ff1a124a77ee271f31895a287670d92b7f4378fe925594b1b1dde1cba7a08d"} err="failed to get container status \"65ff1a124a77ee271f31895a287670d92b7f4378fe925594b1b1dde1cba7a08d\": rpc error: code = NotFound desc = could not find container \"65ff1a124a77ee271f31895a287670d92b7f4378fe925594b1b1dde1cba7a08d\": container with ID starting with 65ff1a124a77ee271f31895a287670d92b7f4378fe925594b1b1dde1cba7a08d not found: ID does not exist" Oct 30 12:48:04.902457 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.902419 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj"] Oct 30 12:48:04.922921 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:04.920013 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-794ct4jsj"] Oct 30 12:48:05.029638 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.029609 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" Oct 30 12:48:05.094856 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.094820 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2cc7ee8-0f74-45aa-944f-71191679478a" path="/var/lib/kubelet/pods/a2cc7ee8-0f74-45aa-944f-71191679478a/volumes" Oct 30 12:48:05.148948 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.148905 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b12aac3-8f71-487d-aba8-9def1e7439a3-kube-api-access-7495z\" (UniqueName: \"kubernetes.io/projected/7b12aac3-8f71-487d-aba8-9def1e7439a3-kube-api-access-7495z\") pod \"7b12aac3-8f71-487d-aba8-9def1e7439a3\" (UID: \"7b12aac3-8f71-487d-aba8-9def1e7439a3\") " Oct 30 12:48:05.149210 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.149002 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b12aac3-8f71-487d-aba8-9def1e7439a3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b12aac3-8f71-487d-aba8-9def1e7439a3-provider-credentials\") pod \"7b12aac3-8f71-487d-aba8-9def1e7439a3\" (UID: \"7b12aac3-8f71-487d-aba8-9def1e7439a3\") " Oct 30 12:48:05.149284 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.149233 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b12aac3-8f71-487d-aba8-9def1e7439a3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b12aac3-8f71-487d-aba8-9def1e7439a3-routing-overlay-volume\") pod \"7b12aac3-8f71-487d-aba8-9def1e7439a3\" (UID: \"7b12aac3-8f71-487d-aba8-9def1e7439a3\") " Oct 30 12:48:05.149284 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.149269 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b12aac3-8f71-487d-aba8-9def1e7439a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b12aac3-8f71-487d-aba8-9def1e7439a3-plugins-config-volume\") pod \"7b12aac3-8f71-487d-aba8-9def1e7439a3\" (UID: \"7b12aac3-8f71-487d-aba8-9def1e7439a3\") " Oct 30 12:48:05.149806 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.149772 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b12aac3-8f71-487d-aba8-9def1e7439a3-routing-overlay-volume" pod "7b12aac3-8f71-487d-aba8-9def1e7439a3" (UID: "7b12aac3-8f71-487d-aba8-9def1e7439a3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:48:05.149925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.149890 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b12aac3-8f71-487d-aba8-9def1e7439a3-plugins-config-volume" pod "7b12aac3-8f71-487d-aba8-9def1e7439a3" (UID: "7b12aac3-8f71-487d-aba8-9def1e7439a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:48:05.150610 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.150582 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b12aac3-8f71-487d-aba8-9def1e7439a3-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:48:05.150704 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.150617 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b12aac3-8f71-487d-aba8-9def1e7439a3-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:48:05.157393 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.157352 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b12aac3-8f71-487d-aba8-9def1e7439a3-kube-api-access-7495z" pod "7b12aac3-8f71-487d-aba8-9def1e7439a3" (UID: "7b12aac3-8f71-487d-aba8-9def1e7439a3"). InnerVolumeSpecName "kube-api-access-7495z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:48:05.163760 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.163719 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b12aac3-8f71-487d-aba8-9def1e7439a3-provider-credentials" pod "7b12aac3-8f71-487d-aba8-9def1e7439a3" (UID: "7b12aac3-8f71-487d-aba8-9def1e7439a3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:48:05.259176 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.258817 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7495z\" (UniqueName: \"kubernetes.io/projected/7b12aac3-8f71-487d-aba8-9def1e7439a3-kube-api-access-7495z\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:48:05.259176 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.258867 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b12aac3-8f71-487d-aba8-9def1e7439a3-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:48:05.840180 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.840129 4468 generic.go:356] "Generic (PLEG): container finished" podID="7b12aac3-8f71-487d-aba8-9def1e7439a3" containerID="06c7307c969337e0264cc06c586b0b6b3ed06142d866b71674aee177211ccca8" exitCode=0 Oct 30 12:48:05.840735 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.840197 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" event={"ID":"7b12aac3-8f71-487d-aba8-9def1e7439a3","Type":"ContainerDied","Data":"06c7307c969337e0264cc06c586b0b6b3ed06142d866b71674aee177211ccca8"} Oct 30 12:48:05.840735 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.840247 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" event={"ID":"7b12aac3-8f71-487d-aba8-9def1e7439a3","Type":"ContainerDied","Data":"eef7aae6a4b227aa7f3e9528f6620c1f75149a637d621050752e35b63489d261"} Oct 30 12:48:05.840735 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.840276 4468 scope.go:122] "RemoveContainer" containerID="06c7307c969337e0264cc06c586b0b6b3ed06142d866b71674aee177211ccca8" Oct 30 12:48:05.840735 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.840312 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb" Oct 30 12:48:05.852299 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.852272 4468 scope.go:122] "RemoveContainer" containerID="06c7307c969337e0264cc06c586b0b6b3ed06142d866b71674aee177211ccca8" Oct 30 12:48:05.852629 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:48:05.852611 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c7307c969337e0264cc06c586b0b6b3ed06142d866b71674aee177211ccca8\": container with ID starting with 06c7307c969337e0264cc06c586b0b6b3ed06142d866b71674aee177211ccca8 not found: ID does not exist" containerID="06c7307c969337e0264cc06c586b0b6b3ed06142d866b71674aee177211ccca8" Oct 30 12:48:05.852670 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.852639 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c7307c969337e0264cc06c586b0b6b3ed06142d866b71674aee177211ccca8"} err="failed to get container status \"06c7307c969337e0264cc06c586b0b6b3ed06142d866b71674aee177211ccca8\": rpc error: code = NotFound desc = could not find container \"06c7307c969337e0264cc06c586b0b6b3ed06142d866b71674aee177211ccca8\": container with ID starting with 06c7307c969337e0264cc06c586b0b6b3ed06142d866b71674aee177211ccca8 not found: ID does not exist" Oct 30 12:48:05.910944 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.910904 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb"] Oct 30 12:48:05.924191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:05.924153 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-7489xzhvb"] Oct 30 12:48:07.092374 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:07.092336 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b12aac3-8f71-487d-aba8-9def1e7439a3" path="/var/lib/kubelet/pods/7b12aac3-8f71-487d-aba8-9def1e7439a3/volumes" Oct 30 12:48:10.713858 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:10.713824 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" Oct 30 12:48:10.797901 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:10.797863 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv"] Oct 30 12:48:15.798298 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:15.798241 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" podUID="8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00" containerName="payload-processing" containerID="cri-o://bae2ceadeee628c3da281d51085ded362239cb36cd2e248a1d1dcb0cd4f38c60" gracePeriod=25 Oct 30 12:48:16.097828 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.097792 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" Oct 30 12:48:16.112879 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.112833 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-provider-credentials\") pod \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\" (UID: \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\") " Oct 30 12:48:16.113114 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.112951 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-plugins-config-volume\") pod \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\" (UID: \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\") " Oct 30 12:48:16.113195 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.113148 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-kube-api-access-pswz6\" (UniqueName: \"kubernetes.io/projected/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-kube-api-access-pswz6\") pod \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\" (UID: \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\") " Oct 30 12:48:16.113256 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.113202 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-routing-overlay-volume\") pod \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\" (UID: \"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00\") " Oct 30 12:48:16.113575 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.113425 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-plugins-config-volume" pod "8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00" (UID: "8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:48:16.113676 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.113607 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-routing-overlay-volume" pod "8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00" (UID: "8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:48:16.113960 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.113934 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:48:16.114086 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.113963 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:48:16.115381 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.115352 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-provider-credentials" pod "8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00" (UID: "8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:48:16.115486 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.115395 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-kube-api-access-pswz6" pod "8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00" (UID: "8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00"). InnerVolumeSpecName "kube-api-access-pswz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:48:16.215807 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.215762 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:48:16.215807 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.215799 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pswz6\" (UniqueName: \"kubernetes.io/projected/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00-kube-api-access-pswz6\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:48:16.924047 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.924012 4468 generic.go:356] "Generic (PLEG): container finished" podID="8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00" containerID="bae2ceadeee628c3da281d51085ded362239cb36cd2e248a1d1dcb0cd4f38c60" exitCode=0 Oct 30 12:48:16.924498 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.924091 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" Oct 30 12:48:16.924498 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.924115 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" event={"ID":"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00","Type":"ContainerDied","Data":"bae2ceadeee628c3da281d51085ded362239cb36cd2e248a1d1dcb0cd4f38c60"} Oct 30 12:48:16.924498 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.924179 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv" event={"ID":"8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00","Type":"ContainerDied","Data":"6a98b8319d5749ca9791e28b040a7f9f31c888dedcc33b9fa6b6d134a69104f8"} Oct 30 12:48:16.924498 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.924208 4468 scope.go:122] "RemoveContainer" containerID="bae2ceadeee628c3da281d51085ded362239cb36cd2e248a1d1dcb0cd4f38c60" Oct 30 12:48:16.939680 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.937322 4468 scope.go:122] "RemoveContainer" containerID="bae2ceadeee628c3da281d51085ded362239cb36cd2e248a1d1dcb0cd4f38c60" Oct 30 12:48:16.939680 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:48:16.938142 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae2ceadeee628c3da281d51085ded362239cb36cd2e248a1d1dcb0cd4f38c60\": container with ID starting with bae2ceadeee628c3da281d51085ded362239cb36cd2e248a1d1dcb0cd4f38c60 not found: ID does not exist" containerID="bae2ceadeee628c3da281d51085ded362239cb36cd2e248a1d1dcb0cd4f38c60" Oct 30 12:48:16.939680 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.938179 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae2ceadeee628c3da281d51085ded362239cb36cd2e248a1d1dcb0cd4f38c60"} err="failed to get container status \"bae2ceadeee628c3da281d51085ded362239cb36cd2e248a1d1dcb0cd4f38c60\": rpc error: code = NotFound desc = could not find container \"bae2ceadeee628c3da281d51085ded362239cb36cd2e248a1d1dcb0cd4f38c60\": container with ID starting with bae2ceadeee628c3da281d51085ded362239cb36cd2e248a1d1dcb0cd4f38c60 not found: ID does not exist" Oct 30 12:48:16.985651 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.985607 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv"] Oct 30 12:48:16.996914 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:16.996874 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574chvcgv"] Oct 30 12:48:17.093002 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:17.092969 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00" path="/var/lib/kubelet/pods/8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00/volumes" Oct 30 12:48:23.601710 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:48:23.601668 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:48:28.393290 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.393244 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp"] Oct 30 12:48:28.394259 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.394236 4468 cpu_manager.go:414] "containerMap: removing container" podUID="a2cc7ee8-0f74-45aa-944f-71191679478a" containerName="payload-processing" Oct 30 12:48:28.394259 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.394259 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2cc7ee8-0f74-45aa-944f-71191679478a" containerName="payload-processing" Oct 30 12:48:28.394428 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.394282 4468 cpu_manager.go:414] "containerMap: removing container" podUID="8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00" containerName="payload-processing" Oct 30 12:48:28.394428 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.394290 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00" containerName="payload-processing" Oct 30 12:48:28.394428 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.394301 4468 cpu_manager.go:414] "containerMap: removing container" podUID="7b12aac3-8f71-487d-aba8-9def1e7439a3" containerName="payload-processing" Oct 30 12:48:28.394428 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.394309 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b12aac3-8f71-487d-aba8-9def1e7439a3" containerName="payload-processing" Oct 30 12:48:28.394650 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.394550 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea3ef2f-83dc-49fb-ad52-c02fdbdfde00" containerName="payload-processing" Oct 30 12:48:28.394650 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.394565 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b12aac3-8f71-487d-aba8-9def1e7439a3" containerName="payload-processing" Oct 30 12:48:28.394650 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.394590 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2cc7ee8-0f74-45aa-944f-71191679478a" containerName="payload-processing" Oct 30 12:48:28.400238 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.400212 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" Oct 30 12:48:28.420256 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.420225 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fedf2cef79c2\"" Oct 30 12:48:28.435939 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.435907 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4ff758d7dea0\"" Oct 30 12:48:28.436235 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.436213 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-4ktzx\"" Oct 30 12:48:28.442420 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.437556 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 12:48:28.445117 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.437617 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 12:48:28.445344 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.438049 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 12:48:28.445471 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.438175 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 12:48:28.446009 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.445975 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp"] Oct 30 12:48:28.549547 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.549501 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/409c8c14-07b1-42da-a6a5-7e4216abcd20-routing-overlay-volume\") pod \"payload-processing-external-model-869554dfbc-tvkrp\" (UID: \"409c8c14-07b1-42da-a6a5-7e4216abcd20\") " pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" Oct 30 12:48:28.549743 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.549663 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/409c8c14-07b1-42da-a6a5-7e4216abcd20-provider-credentials\") pod \"payload-processing-external-model-869554dfbc-tvkrp\" (UID: \"409c8c14-07b1-42da-a6a5-7e4216abcd20\") " pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" Oct 30 12:48:28.549743 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.549716 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409c8c14-07b1-42da-a6a5-7e4216abcd20-plugins-config-volume\") pod \"payload-processing-external-model-869554dfbc-tvkrp\" (UID: \"409c8c14-07b1-42da-a6a5-7e4216abcd20\") " pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" Oct 30 12:48:28.549845 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.549779 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxgnw\" (UniqueName: \"kubernetes.io/projected/409c8c14-07b1-42da-a6a5-7e4216abcd20-kube-api-access-sxgnw\") pod \"payload-processing-external-model-869554dfbc-tvkrp\" (UID: \"409c8c14-07b1-42da-a6a5-7e4216abcd20\") " pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" Oct 30 12:48:28.650997 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.650914 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxgnw\" (UniqueName: \"kubernetes.io/projected/409c8c14-07b1-42da-a6a5-7e4216abcd20-kube-api-access-sxgnw\") pod \"payload-processing-external-model-869554dfbc-tvkrp\" (UID: \"409c8c14-07b1-42da-a6a5-7e4216abcd20\") " pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" Oct 30 12:48:28.651209 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.651017 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/409c8c14-07b1-42da-a6a5-7e4216abcd20-routing-overlay-volume\") pod \"payload-processing-external-model-869554dfbc-tvkrp\" (UID: \"409c8c14-07b1-42da-a6a5-7e4216abcd20\") " pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" Oct 30 12:48:28.651209 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.651114 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/409c8c14-07b1-42da-a6a5-7e4216abcd20-provider-credentials\") pod \"payload-processing-external-model-869554dfbc-tvkrp\" (UID: \"409c8c14-07b1-42da-a6a5-7e4216abcd20\") " pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" Oct 30 12:48:28.651209 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.651166 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409c8c14-07b1-42da-a6a5-7e4216abcd20-plugins-config-volume\") pod \"payload-processing-external-model-869554dfbc-tvkrp\" (UID: \"409c8c14-07b1-42da-a6a5-7e4216abcd20\") " pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" Oct 30 12:48:28.651771 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.651745 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/409c8c14-07b1-42da-a6a5-7e4216abcd20-routing-overlay-volume\") pod \"payload-processing-external-model-869554dfbc-tvkrp\" (UID: \"409c8c14-07b1-42da-a6a5-7e4216abcd20\") " pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" Oct 30 12:48:28.651898 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.651877 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409c8c14-07b1-42da-a6a5-7e4216abcd20-plugins-config-volume\") pod \"payload-processing-external-model-869554dfbc-tvkrp\" (UID: \"409c8c14-07b1-42da-a6a5-7e4216abcd20\") " pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" Oct 30 12:48:28.653872 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.653839 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/409c8c14-07b1-42da-a6a5-7e4216abcd20-provider-credentials\") pod \"payload-processing-external-model-869554dfbc-tvkrp\" (UID: \"409c8c14-07b1-42da-a6a5-7e4216abcd20\") " pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" Oct 30 12:48:28.677035 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.676999 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxgnw\" (UniqueName: \"kubernetes.io/projected/409c8c14-07b1-42da-a6a5-7e4216abcd20-kube-api-access-sxgnw\") pod \"payload-processing-external-model-869554dfbc-tvkrp\" (UID: \"409c8c14-07b1-42da-a6a5-7e4216abcd20\") " pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" Oct 30 12:48:28.752220 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:28.752177 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" Oct 30 12:48:29.079416 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:48:29.079373 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409c8c14_07b1_42da_a6a5_7e4216abcd20.slice/crio-83012bd8d130edb98d58b6a84899003570e103619238fd425c1bbe3c9a2e3025 WatchSource:0}: Error finding container 83012bd8d130edb98d58b6a84899003570e103619238fd425c1bbe3c9a2e3025: Status 404 returned error can't find the container with id 83012bd8d130edb98d58b6a84899003570e103619238fd425c1bbe3c9a2e3025 Oct 30 12:48:29.095167 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:29.095129 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp"] Oct 30 12:48:30.045586 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:30.045547 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" event={"ID":"409c8c14-07b1-42da-a6a5-7e4216abcd20","Type":"ContainerStarted","Data":"0f78c4123e904f5bdefbcb21fe96c681738ba0f4849c3ea14389baa299d68b53"} Oct 30 12:48:30.046021 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:30.045607 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" event={"ID":"409c8c14-07b1-42da-a6a5-7e4216abcd20","Type":"ContainerStarted","Data":"83012bd8d130edb98d58b6a84899003570e103619238fd425c1bbe3c9a2e3025"} Oct 30 12:48:30.046021 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:30.045636 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" Oct 30 12:48:30.098886 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:30.098817 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" podStartSLOduration=2.098797065 podStartE2EDuration="2.098797065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:30.094563163 +0000 UTC m=+3261.128701559" watchObservedRunningTime="2026-10-30 12:48:30.098797065 +0000 UTC m=+3261.132935463" Oct 30 12:48:41.058836 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:41.058805 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" Oct 30 12:48:46.712467 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.712420 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj"] Oct 30 12:48:46.732018 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.731985 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.740468 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.740431 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj"] Oct 30 12:48:46.751565 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.751534 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lkcrv\"" Oct 30 12:48:46.832107 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.832031 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rrtl\" (UniqueName: \"kubernetes.io/projected/16dae47c-220c-4d83-89e5-52d1746775e8-kube-api-access-9rrtl\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.832329 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.832266 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.832391 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.832370 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.832441 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.832418 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dae47c-220c-4d83-89e5-52d1746775e8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.832509 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.832442 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.832627 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.832610 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16dae47c-220c-4d83-89e5-52d1746775e8-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.832693 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.832641 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.832754 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.832723 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.832815 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.832770 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dae47c-220c-4d83-89e5-52d1746775e8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.933823 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.933786 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.934013 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.933843 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.934013 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.933887 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dae47c-220c-4d83-89e5-52d1746775e8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.934013 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.933916 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.934013 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.933970 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16dae47c-220c-4d83-89e5-52d1746775e8-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.934013 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.933996 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.934320 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.934023 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.934320 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.934051 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dae47c-220c-4d83-89e5-52d1746775e8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.934320 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.934187 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rrtl\" (UniqueName: \"kubernetes.io/projected/16dae47c-220c-4d83-89e5-52d1746775e8-kube-api-access-9rrtl\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.934479 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.934320 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.934715 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.934690 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dae47c-220c-4d83-89e5-52d1746775e8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.934838 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.934814 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.934906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.934857 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.934906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.934881 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.936174 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.936153 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.936790 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.936771 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dae47c-220c-4d83-89e5-52d1746775e8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.958830 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.958793 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16dae47c-220c-4d83-89e5-52d1746775e8-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:46.959288 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:46.959262 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rrtl\" (UniqueName: \"kubernetes.io/projected/16dae47c-220c-4d83-89e5-52d1746775e8-kube-api-access-9rrtl\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qtfcj\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:47.061108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:47.060973 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:47.388520 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:48:47.388489 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16dae47c_220c_4d83_89e5_52d1746775e8.slice/crio-903900eb4b6cbc870e30a3d9e94a45590f3cb2beed6c7f6463318f362144064d WatchSource:0}: Error finding container 903900eb4b6cbc870e30a3d9e94a45590f3cb2beed6c7f6463318f362144064d: Status 404 returned error can't find the container with id 903900eb4b6cbc870e30a3d9e94a45590f3cb2beed6c7f6463318f362144064d Oct 30 12:48:47.391492 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:47.391462 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:48:47.391573 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:47.391547 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:48:47.391614 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:47.391578 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:48:47.392016 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:47.391992 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj"] Oct 30 12:48:48.193320 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:48.193283 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" event={"ID":"16dae47c-220c-4d83-89e5-52d1746775e8","Type":"ContainerStarted","Data":"7cbeeb3b2b35a7e456f12b1594e679754a5929d5a07785c074026e1e873caf71"} Oct 30 12:48:48.193752 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:48.193343 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" event={"ID":"16dae47c-220c-4d83-89e5-52d1746775e8","Type":"ContainerStarted","Data":"903900eb4b6cbc870e30a3d9e94a45590f3cb2beed6c7f6463318f362144064d"} Oct 30 12:48:48.281716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:48.281646 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" podStartSLOduration=2.281627823 podStartE2EDuration="2.281627823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:48.266619925 +0000 UTC m=+3279.300758321" watchObservedRunningTime="2026-10-30 12:48:48.281627823 +0000 UTC m=+3279.315766220" Oct 30 12:48:49.061872 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:49.061834 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:49.063928 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:49.063898 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 12:48:49.064082 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:49.063952 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" podUID="16dae47c-220c-4d83-89e5-52d1746775e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 12:48:50.062146 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:50.062109 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 12:48:50.062672 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:50.062173 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" podUID="16dae47c-220c-4d83-89e5-52d1746775e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 12:48:51.062201 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:51.062105 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 12:48:51.062201 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:51.062175 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" podUID="16dae47c-220c-4d83-89e5-52d1746775e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 12:48:52.061996 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:52.061959 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 12:48:52.062193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:52.062017 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" podUID="16dae47c-220c-4d83-89e5-52d1746775e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 12:48:53.062095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:53.062036 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 12:48:53.062515 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:53.062122 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" podUID="16dae47c-220c-4d83-89e5-52d1746775e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 12:48:53.602237 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:48:53.602180 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:48:54.062410 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:54.062363 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 12:48:54.062843 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:54.062440 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" podUID="16dae47c-220c-4d83-89e5-52d1746775e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 12:48:55.061590 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:55.061552 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 12:48:55.061787 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:55.061618 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" podUID="16dae47c-220c-4d83-89e5-52d1746775e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 12:48:56.140047 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:56.140013 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:56.140538 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:56.140455 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:48:56.141253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:48:56.141238 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 12:49:01.769820 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.769775 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq"] Oct 30 12:49:01.774906 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.774880 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.798464 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.798423 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq"] Oct 30 12:49:01.886420 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.886379 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6305f49-03f1-4d47-a0a7-19776c672543-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.886420 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.886419 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6305f49-03f1-4d47-a0a7-19776c672543-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.886675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.886439 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6305f49-03f1-4d47-a0a7-19776c672543-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.886675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.886553 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7987\" (UniqueName: \"kubernetes.io/projected/d6305f49-03f1-4d47-a0a7-19776c672543-kube-api-access-z7987\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.886675 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.886619 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6305f49-03f1-4d47-a0a7-19776c672543-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.886849 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.886738 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6305f49-03f1-4d47-a0a7-19776c672543-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.886950 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.886933 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6305f49-03f1-4d47-a0a7-19776c672543-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.887013 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.886975 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6305f49-03f1-4d47-a0a7-19776c672543-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.887136 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.887120 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6305f49-03f1-4d47-a0a7-19776c672543-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.988014 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.987972 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6305f49-03f1-4d47-a0a7-19776c672543-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.988264 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.988023 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6305f49-03f1-4d47-a0a7-19776c672543-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.988264 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.988113 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6305f49-03f1-4d47-a0a7-19776c672543-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.988264 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.988157 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6305f49-03f1-4d47-a0a7-19776c672543-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.988264 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.988186 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6305f49-03f1-4d47-a0a7-19776c672543-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.988264 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.988211 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6305f49-03f1-4d47-a0a7-19776c672543-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.988264 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.988249 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7987\" (UniqueName: \"kubernetes.io/projected/d6305f49-03f1-4d47-a0a7-19776c672543-kube-api-access-z7987\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.988587 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.988318 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6305f49-03f1-4d47-a0a7-19776c672543-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.988587 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.988416 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6305f49-03f1-4d47-a0a7-19776c672543-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.988718 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.988633 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6305f49-03f1-4d47-a0a7-19776c672543-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.988828 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.988796 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6305f49-03f1-4d47-a0a7-19776c672543-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.989198 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.989179 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6305f49-03f1-4d47-a0a7-19776c672543-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.989363 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.989336 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6305f49-03f1-4d47-a0a7-19776c672543-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.989448 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.989373 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6305f49-03f1-4d47-a0a7-19776c672543-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.990660 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.990636 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6305f49-03f1-4d47-a0a7-19776c672543-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:01.990909 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:01.990890 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6305f49-03f1-4d47-a0a7-19776c672543-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:02.013032 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:02.013000 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6305f49-03f1-4d47-a0a7-19776c672543-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:02.013392 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:02.013369 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7987\" (UniqueName: \"kubernetes.io/projected/d6305f49-03f1-4d47-a0a7-19776c672543-kube-api-access-z7987\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7tbjq\" (UID: \"d6305f49-03f1-4d47-a0a7-19776c672543\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:02.093911 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:02.093828 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:02.421441 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:49:02.421402 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6305f49_03f1_4d47_a0a7_19776c672543.slice/crio-07bd62ae714c3becc0fa464958fa9487769506ed7957d3e1f40698ca9cac4f3f WatchSource:0}: Error finding container 07bd62ae714c3becc0fa464958fa9487769506ed7957d3e1f40698ca9cac4f3f: Status 404 returned error can't find the container with id 07bd62ae714c3becc0fa464958fa9487769506ed7957d3e1f40698ca9cac4f3f Oct 30 12:49:02.424829 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:02.424802 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq"] Oct 30 12:49:02.425184 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:02.425155 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:49:02.425240 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:02.425218 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:49:02.425274 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:02.425247 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:49:03.345338 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:03.345297 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" event={"ID":"d6305f49-03f1-4d47-a0a7-19776c672543","Type":"ContainerStarted","Data":"564652ea5a3f7a2f47952f8d431c953da08050608a25bf26181d87030fd01a6e"} Oct 30 12:49:03.345919 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:03.345362 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" event={"ID":"d6305f49-03f1-4d47-a0a7-19776c672543","Type":"ContainerStarted","Data":"07bd62ae714c3becc0fa464958fa9487769506ed7957d3e1f40698ca9cac4f3f"} Oct 30 12:49:03.434802 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:03.434734 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" podStartSLOduration=2.4347167450000002 podStartE2EDuration="2.434716745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:03.422742915 +0000 UTC m=+3294.456881312" watchObservedRunningTime="2026-10-30 12:49:03.434716745 +0000 UTC m=+3294.468855140" Oct 30 12:49:04.094645 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:04.094609 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:04.096783 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:04.096753 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 12:49:04.096955 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:04.096829 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" podUID="d6305f49-03f1-4d47-a0a7-19776c672543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 12:49:05.094997 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:05.094957 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 12:49:05.095488 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:05.095015 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" podUID="d6305f49-03f1-4d47-a0a7-19776c672543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 12:49:06.094249 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:06.094214 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 12:49:06.094579 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:06.094297 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" podUID="d6305f49-03f1-4d47-a0a7-19776c672543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 12:49:07.094604 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:07.094561 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 12:49:07.095036 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:07.094634 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" podUID="d6305f49-03f1-4d47-a0a7-19776c672543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 12:49:08.094680 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:08.094625 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 12:49:08.095246 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:08.094694 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" podUID="d6305f49-03f1-4d47-a0a7-19776c672543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 12:49:09.095185 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:09.095153 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 12:49:09.095597 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:09.095213 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" podUID="d6305f49-03f1-4d47-a0a7-19776c672543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 12:49:10.094548 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:10.094501 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 12:49:10.094753 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:10.094595 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" podUID="d6305f49-03f1-4d47-a0a7-19776c672543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 12:49:11.279280 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:11.279240 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:11.279966 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:11.279944 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:11.281419 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:11.281394 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7tbjq" Oct 30 12:49:23.602522 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:49:23.602482 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:49:31.914611 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:31.914554 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt"] Oct 30 12:49:31.920201 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:31.920174 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:31.951028 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:31.950988 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt"] Oct 30 12:49:32.041625 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.041591 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrt2p\" (UniqueName: \"kubernetes.io/projected/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-kube-api-access-rrt2p\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.041835 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.041657 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.041908 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.041879 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.041975 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.041955 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.042135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.042112 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.042217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.042197 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.042289 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.042276 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.042429 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.042410 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.042470 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.042440 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.144335 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.144281 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.144555 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.144370 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.144555 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.144430 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.144555 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.144470 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.144748 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.144557 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.144748 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.144612 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.144748 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.144636 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.144748 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.144678 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrt2p\" (UniqueName: \"kubernetes.io/projected/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-kube-api-access-rrt2p\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.144748 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.144735 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.145037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.144738 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.145037 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.144991 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.145201 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.145175 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.145449 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.145427 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.145546 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.145525 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.147099 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.147061 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.147288 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.147269 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.171881 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.171808 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.172104 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.172062 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrt2p\" (UniqueName: \"kubernetes.io/projected/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-kube-api-access-rrt2p\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kxnqt\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.239712 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.239668 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:32.566623 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:49:32.566592 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60e79ec_6b0e_4725_81f3_d02c72fc8efb.slice/crio-8b81536df49c05389a635023394682cda9241741585b9d92847de617f19d1c0b WatchSource:0}: Error finding container 8b81536df49c05389a635023394682cda9241741585b9d92847de617f19d1c0b: Status 404 returned error can't find the container with id 8b81536df49c05389a635023394682cda9241741585b9d92847de617f19d1c0b Oct 30 12:49:32.569148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.569123 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt"] Oct 30 12:49:32.569496 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.569466 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:49:32.569545 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.569534 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:49:32.569591 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.569566 4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:49:32.639153 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:32.639047 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" event={"ID":"b60e79ec-6b0e-4725-81f3-d02c72fc8efb","Type":"ContainerStarted","Data":"8b81536df49c05389a635023394682cda9241741585b9d92847de617f19d1c0b"} Oct 30 12:49:33.650650 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:33.650614 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" event={"ID":"b60e79ec-6b0e-4725-81f3-d02c72fc8efb","Type":"ContainerStarted","Data":"0e5f3ecb5eb7880bc204bb9e9feaa5f9f3f6ae730510603e65e48f86032af4f5"} Oct 30 12:49:33.754094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:33.753563 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" podStartSLOduration=2.753537083 podStartE2EDuration="2.753537083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:33.720474745 +0000 UTC m=+3324.754613142" watchObservedRunningTime="2026-10-30 12:49:33.753537083 +0000 UTC m=+3324.787675480" Oct 30 12:49:34.240638 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:34.240593 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:34.243042 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:34.243012 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 12:49:34.243253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:34.243102 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" podUID="b60e79ec-6b0e-4725-81f3-d02c72fc8efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 12:49:35.240818 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:35.240778 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 12:49:35.241342 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:35.240849 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" podUID="b60e79ec-6b0e-4725-81f3-d02c72fc8efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 12:49:36.240680 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:36.240637 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 12:49:36.240892 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:36.240706 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" podUID="b60e79ec-6b0e-4725-81f3-d02c72fc8efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 12:49:37.240545 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:37.240508 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 12:49:37.240836 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:37.240563 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" podUID="b60e79ec-6b0e-4725-81f3-d02c72fc8efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 12:49:38.240229 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:38.240188 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 12:49:38.240640 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:38.240257 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" podUID="b60e79ec-6b0e-4725-81f3-d02c72fc8efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 12:49:39.241125 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:39.241079 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 12:49:39.241557 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:39.241165 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" podUID="b60e79ec-6b0e-4725-81f3-d02c72fc8efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 12:49:40.241169 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:40.241125 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 12:49:40.241587 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:40.241212 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" podUID="b60e79ec-6b0e-4725-81f3-d02c72fc8efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 12:49:42.240639 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:42.240583 4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:49:42.241176 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:42.240684 4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" podUID="b60e79ec-6b0e-4725-81f3-d02c72fc8efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:49:42.241993 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:42.241970 4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:42.242540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:42.242524 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:42.243335 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:49:42.243311 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 12:49:53.605894 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:49:53.605855 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:50:11.903961 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:11.903922 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l"] Oct 30 12:50:11.908633 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:11.908608 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" Oct 30 12:50:11.927742 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:11.927707 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l"] Oct 30 12:50:12.087193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:12.087150 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/488d90dc-7ac7-41fc-bf59-64d3d09732c9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l\" (UID: \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" Oct 30 12:50:12.087389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:12.087201 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/488d90dc-7ac7-41fc-bf59-64d3d09732c9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l\" (UID: \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" Oct 30 12:50:12.087389 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:12.087299 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njffj\" (UniqueName: \"kubernetes.io/projected/488d90dc-7ac7-41fc-bf59-64d3d09732c9-kube-api-access-njffj\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l\" (UID: \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" Oct 30 12:50:12.087488 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:12.087464 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/488d90dc-7ac7-41fc-bf59-64d3d09732c9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l\" (UID: \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" Oct 30 12:50:12.189196 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:12.189153 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/488d90dc-7ac7-41fc-bf59-64d3d09732c9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l\" (UID: \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" Oct 30 12:50:12.189391 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:12.189295 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/488d90dc-7ac7-41fc-bf59-64d3d09732c9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l\" (UID: \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" Oct 30 12:50:12.189391 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:12.189322 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/488d90dc-7ac7-41fc-bf59-64d3d09732c9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l\" (UID: \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" Oct 30 12:50:12.189391 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:12.189361 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njffj\" (UniqueName: \"kubernetes.io/projected/488d90dc-7ac7-41fc-bf59-64d3d09732c9-kube-api-access-njffj\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l\" (UID: \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" Oct 30 12:50:12.190227 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:12.190193 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/488d90dc-7ac7-41fc-bf59-64d3d09732c9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l\" (UID: \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" Oct 30 12:50:12.190369 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:12.190269 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/488d90dc-7ac7-41fc-bf59-64d3d09732c9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l\" (UID: \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" Oct 30 12:50:12.191976 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:12.191949 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/488d90dc-7ac7-41fc-bf59-64d3d09732c9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l\" (UID: \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" Oct 30 12:50:12.214384 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:12.214351 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njffj\" (UniqueName: \"kubernetes.io/projected/488d90dc-7ac7-41fc-bf59-64d3d09732c9-kube-api-access-njffj\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l\" (UID: \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" Oct 30 12:50:12.227336 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:12.227304 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" Oct 30 12:50:12.569474 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:50:12.569423 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488d90dc_7ac7_41fc_bf59_64d3d09732c9.slice/crio-6fe13c71aaf999074ac5555a1223cc7b49259723d59677eeec8226e016dd1eb5 WatchSource:0}: Error finding container 6fe13c71aaf999074ac5555a1223cc7b49259723d59677eeec8226e016dd1eb5: Status 404 returned error can't find the container with id 6fe13c71aaf999074ac5555a1223cc7b49259723d59677eeec8226e016dd1eb5 Oct 30 12:50:12.572166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:12.572140 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l"] Oct 30 12:50:13.141228 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:13.141188 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" event={"ID":"488d90dc-7ac7-41fc-bf59-64d3d09732c9","Type":"ContainerStarted","Data":"59d1087e2a6038fdbf00d70ce153759bf06bad0447f9da2cf33acfa08723b6a6"} Oct 30 12:50:13.141666 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:13.141256 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" Oct 30 12:50:13.141666 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:13.141279 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" event={"ID":"488d90dc-7ac7-41fc-bf59-64d3d09732c9","Type":"ContainerStarted","Data":"6fe13c71aaf999074ac5555a1223cc7b49259723d59677eeec8226e016dd1eb5"} Oct 30 12:50:13.189243 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:13.189182 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" podStartSLOduration=2.189167125 podStartE2EDuration="2.189167125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:13.184850113 +0000 UTC m=+3364.218988509" watchObservedRunningTime="2026-10-30 12:50:13.189167125 +0000 UTC m=+3364.223305522" Oct 30 12:50:23.372671 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:23.372444 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l"] Oct 30 12:50:23.381643 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:23.381609 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" Oct 30 12:50:23.671848 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:50:23.671803 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:50:24.541791 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:24.541746 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd"] Oct 30 12:50:28.374372 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:28.374302 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" podUID="488d90dc-7ac7-41fc-bf59-64d3d09732c9" containerName="payload-processing" containerID="cri-o://59d1087e2a6038fdbf00d70ce153759bf06bad0447f9da2cf33acfa08723b6a6" gracePeriod=25 Oct 30 12:50:28.730100 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:50:28.722963 4468 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488d90dc_7ac7_41fc_bf59_64d3d09732c9.slice/crio-conmon-59d1087e2a6038fdbf00d70ce153759bf06bad0447f9da2cf33acfa08723b6a6.scope\": RecentStats: unable to find data in memory cache]" Oct 30 12:50:29.340225 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.340114 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" Oct 30 12:50:29.346436 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.346391 4468 generic.go:356] "Generic (PLEG): container finished" podID="488d90dc-7ac7-41fc-bf59-64d3d09732c9" containerID="59d1087e2a6038fdbf00d70ce153759bf06bad0447f9da2cf33acfa08723b6a6" exitCode=0 Oct 30 12:50:29.347861 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.346712 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" Oct 30 12:50:29.351298 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.346744 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" event={"ID":"488d90dc-7ac7-41fc-bf59-64d3d09732c9","Type":"ContainerDied","Data":"59d1087e2a6038fdbf00d70ce153759bf06bad0447f9da2cf33acfa08723b6a6"} Oct 30 12:50:29.351548 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.351524 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l" event={"ID":"488d90dc-7ac7-41fc-bf59-64d3d09732c9","Type":"ContainerDied","Data":"6fe13c71aaf999074ac5555a1223cc7b49259723d59677eeec8226e016dd1eb5"} Oct 30 12:50:29.351716 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.351668 4468 scope.go:122] "RemoveContainer" containerID="59d1087e2a6038fdbf00d70ce153759bf06bad0447f9da2cf33acfa08723b6a6" Oct 30 12:50:29.417095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.412551 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/488d90dc-7ac7-41fc-bf59-64d3d09732c9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/488d90dc-7ac7-41fc-bf59-64d3d09732c9-routing-overlay-volume\") pod \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\" (UID: \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\") " Oct 30 12:50:29.417095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.412710 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/488d90dc-7ac7-41fc-bf59-64d3d09732c9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/488d90dc-7ac7-41fc-bf59-64d3d09732c9-provider-credentials\") pod \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\" (UID: \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\") " Oct 30 12:50:29.417095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.412784 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/488d90dc-7ac7-41fc-bf59-64d3d09732c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/488d90dc-7ac7-41fc-bf59-64d3d09732c9-plugins-config-volume\") pod \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\" (UID: \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\") " Oct 30 12:50:29.417095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.412819 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/488d90dc-7ac7-41fc-bf59-64d3d09732c9-kube-api-access-njffj\" (UniqueName: \"kubernetes.io/projected/488d90dc-7ac7-41fc-bf59-64d3d09732c9-kube-api-access-njffj\") pod \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\" (UID: \"488d90dc-7ac7-41fc-bf59-64d3d09732c9\") " Oct 30 12:50:29.450265 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.446470 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488d90dc-7ac7-41fc-bf59-64d3d09732c9-routing-overlay-volume" pod "488d90dc-7ac7-41fc-bf59-64d3d09732c9" (UID: "488d90dc-7ac7-41fc-bf59-64d3d09732c9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:29.450265 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.446883 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488d90dc-7ac7-41fc-bf59-64d3d09732c9-plugins-config-volume" pod "488d90dc-7ac7-41fc-bf59-64d3d09732c9" (UID: "488d90dc-7ac7-41fc-bf59-64d3d09732c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:29.497804 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.497650 4468 scope.go:122] "RemoveContainer" containerID="59d1087e2a6038fdbf00d70ce153759bf06bad0447f9da2cf33acfa08723b6a6" Oct 30 12:50:29.498476 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.498443 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488d90dc-7ac7-41fc-bf59-64d3d09732c9-provider-credentials" pod "488d90dc-7ac7-41fc-bf59-64d3d09732c9" (UID: "488d90dc-7ac7-41fc-bf59-64d3d09732c9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:29.498596 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:50:29.498567 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59d1087e2a6038fdbf00d70ce153759bf06bad0447f9da2cf33acfa08723b6a6\": container with ID starting with 59d1087e2a6038fdbf00d70ce153759bf06bad0447f9da2cf33acfa08723b6a6 not found: ID does not exist" containerID="59d1087e2a6038fdbf00d70ce153759bf06bad0447f9da2cf33acfa08723b6a6" Oct 30 12:50:29.498668 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.498607 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59d1087e2a6038fdbf00d70ce153759bf06bad0447f9da2cf33acfa08723b6a6"} err="failed to get container status \"59d1087e2a6038fdbf00d70ce153759bf06bad0447f9da2cf33acfa08723b6a6\": rpc error: code = NotFound desc = could not find container \"59d1087e2a6038fdbf00d70ce153759bf06bad0447f9da2cf33acfa08723b6a6\": container with ID starting with 59d1087e2a6038fdbf00d70ce153759bf06bad0447f9da2cf33acfa08723b6a6 not found: ID does not exist" Oct 30 12:50:29.499380 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.499266 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488d90dc-7ac7-41fc-bf59-64d3d09732c9-kube-api-access-njffj" pod "488d90dc-7ac7-41fc-bf59-64d3d09732c9" (UID: "488d90dc-7ac7-41fc-bf59-64d3d09732c9"). InnerVolumeSpecName "kube-api-access-njffj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:29.522103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.520292 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/488d90dc-7ac7-41fc-bf59-64d3d09732c9-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:50:29.522103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.520342 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/488d90dc-7ac7-41fc-bf59-64d3d09732c9-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:50:29.522103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.520358 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/488d90dc-7ac7-41fc-bf59-64d3d09732c9-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:50:29.522103 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.520373 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njffj\" (UniqueName: \"kubernetes.io/projected/488d90dc-7ac7-41fc-bf59-64d3d09732c9-kube-api-access-njffj\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:50:29.546344 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.546284 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" podUID="ef130a03-b77d-4f73-a975-33f09df99896" containerName="payload-processing" containerID="cri-o://10e5ab60e0a09298387f37853014477e4edc4ce5d14edef3352cf2a32c37db2c" gracePeriod=25 Oct 30 12:50:29.800150 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.795428 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l"] Oct 30 12:50:29.801486 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:29.801160 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-574cplq4l"] Oct 30 12:50:30.283178 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.282733 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" Oct 30 12:50:30.358216 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.358175 4468 generic.go:356] "Generic (PLEG): container finished" podID="ef130a03-b77d-4f73-a975-33f09df99896" containerID="10e5ab60e0a09298387f37853014477e4edc4ce5d14edef3352cf2a32c37db2c" exitCode=0 Oct 30 12:50:30.358432 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.358233 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" Oct 30 12:50:30.358432 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.358265 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" event={"ID":"ef130a03-b77d-4f73-a975-33f09df99896","Type":"ContainerDied","Data":"10e5ab60e0a09298387f37853014477e4edc4ce5d14edef3352cf2a32c37db2c"} Oct 30 12:50:30.358432 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.358326 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd" event={"ID":"ef130a03-b77d-4f73-a975-33f09df99896","Type":"ContainerDied","Data":"b6eb453b354ffe27a5281417ad75ed184e2d0b0e1bc003db16fad2ca86e4fc58"} Oct 30 12:50:30.358432 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.358358 4468 scope.go:122] "RemoveContainer" containerID="10e5ab60e0a09298387f37853014477e4edc4ce5d14edef3352cf2a32c37db2c" Oct 30 12:50:30.369728 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.369701 4468 scope.go:122] "RemoveContainer" containerID="10e5ab60e0a09298387f37853014477e4edc4ce5d14edef3352cf2a32c37db2c" Oct 30 12:50:30.370132 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:50:30.370110 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e5ab60e0a09298387f37853014477e4edc4ce5d14edef3352cf2a32c37db2c\": container with ID starting with 10e5ab60e0a09298387f37853014477e4edc4ce5d14edef3352cf2a32c37db2c not found: ID does not exist" containerID="10e5ab60e0a09298387f37853014477e4edc4ce5d14edef3352cf2a32c37db2c" Oct 30 12:50:30.370239 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.370140 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e5ab60e0a09298387f37853014477e4edc4ce5d14edef3352cf2a32c37db2c"} err="failed to get container status \"10e5ab60e0a09298387f37853014477e4edc4ce5d14edef3352cf2a32c37db2c\": rpc error: code = NotFound desc = could not find container \"10e5ab60e0a09298387f37853014477e4edc4ce5d14edef3352cf2a32c37db2c\": container with ID starting with 10e5ab60e0a09298387f37853014477e4edc4ce5d14edef3352cf2a32c37db2c not found: ID does not exist" Oct 30 12:50:30.391355 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.391317 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef130a03-b77d-4f73-a975-33f09df99896-kube-api-access-n8lpf\" (UniqueName: \"kubernetes.io/projected/ef130a03-b77d-4f73-a975-33f09df99896-kube-api-access-n8lpf\") pod \"ef130a03-b77d-4f73-a975-33f09df99896\" (UID: \"ef130a03-b77d-4f73-a975-33f09df99896\") " Oct 30 12:50:30.392288 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.392261 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef130a03-b77d-4f73-a975-33f09df99896-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef130a03-b77d-4f73-a975-33f09df99896-routing-overlay-volume\") pod \"ef130a03-b77d-4f73-a975-33f09df99896\" (UID: \"ef130a03-b77d-4f73-a975-33f09df99896\") " Oct 30 12:50:30.392540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.392473 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef130a03-b77d-4f73-a975-33f09df99896-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef130a03-b77d-4f73-a975-33f09df99896-plugins-config-volume\") pod \"ef130a03-b77d-4f73-a975-33f09df99896\" (UID: \"ef130a03-b77d-4f73-a975-33f09df99896\") " Oct 30 12:50:30.392671 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.392606 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef130a03-b77d-4f73-a975-33f09df99896-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef130a03-b77d-4f73-a975-33f09df99896-provider-credentials\") pod \"ef130a03-b77d-4f73-a975-33f09df99896\" (UID: \"ef130a03-b77d-4f73-a975-33f09df99896\") " Oct 30 12:50:30.392671 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.392638 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef130a03-b77d-4f73-a975-33f09df99896-routing-overlay-volume" pod "ef130a03-b77d-4f73-a975-33f09df99896" (UID: "ef130a03-b77d-4f73-a975-33f09df99896"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:30.393031 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.393002 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef130a03-b77d-4f73-a975-33f09df99896-plugins-config-volume" pod "ef130a03-b77d-4f73-a975-33f09df99896" (UID: "ef130a03-b77d-4f73-a975-33f09df99896"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:30.393277 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.393259 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef130a03-b77d-4f73-a975-33f09df99896-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:50:30.393339 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.393283 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef130a03-b77d-4f73-a975-33f09df99896-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:50:30.394349 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.394327 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef130a03-b77d-4f73-a975-33f09df99896-kube-api-access-n8lpf" pod "ef130a03-b77d-4f73-a975-33f09df99896" (UID: "ef130a03-b77d-4f73-a975-33f09df99896"). InnerVolumeSpecName "kube-api-access-n8lpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:30.394787 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.394768 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef130a03-b77d-4f73-a975-33f09df99896-provider-credentials" pod "ef130a03-b77d-4f73-a975-33f09df99896" (UID: "ef130a03-b77d-4f73-a975-33f09df99896"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:30.494955 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.494915 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef130a03-b77d-4f73-a975-33f09df99896-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:50:30.494955 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.494948 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8lpf\" (UniqueName: \"kubernetes.io/projected/ef130a03-b77d-4f73-a975-33f09df99896-kube-api-access-n8lpf\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:50:30.719558 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.719521 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd"] Oct 30 12:50:30.732502 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:30.732455 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-59dbp85qd"] Oct 30 12:50:31.091907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:31.091764 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="488d90dc-7ac7-41fc-bf59-64d3d09732c9" path="/var/lib/kubelet/pods/488d90dc-7ac7-41fc-bf59-64d3d09732c9/volumes" Oct 30 12:50:31.093110 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:31.092560 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef130a03-b77d-4f73-a975-33f09df99896" path="/var/lib/kubelet/pods/ef130a03-b77d-4f73-a975-33f09df99896/volumes" Oct 30 12:50:40.469634 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.469597 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-99c554458-p54jm"] Oct 30 12:50:40.470436 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.470420 4468 cpu_manager.go:414] "containerMap: removing container" podUID="488d90dc-7ac7-41fc-bf59-64d3d09732c9" containerName="payload-processing" Oct 30 12:50:40.470506 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.470437 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="488d90dc-7ac7-41fc-bf59-64d3d09732c9" containerName="payload-processing" Oct 30 12:50:40.470506 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.470474 4468 cpu_manager.go:414] "containerMap: removing container" podUID="ef130a03-b77d-4f73-a975-33f09df99896" containerName="payload-processing" Oct 30 12:50:40.470506 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.470479 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef130a03-b77d-4f73-a975-33f09df99896" containerName="payload-processing" Oct 30 12:50:40.470604 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.470584 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef130a03-b77d-4f73-a975-33f09df99896" containerName="payload-processing" Oct 30 12:50:40.470604 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.470602 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="488d90dc-7ac7-41fc-bf59-64d3d09732c9" containerName="payload-processing" Oct 30 12:50:40.476241 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.476220 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" Oct 30 12:50:40.494687 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.494655 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-99c554458-p54jm"] Oct 30 12:50:40.497093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.496724 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-57f1b32b361d\"" Oct 30 12:50:40.585799 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.585740 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c51358-a619-47d6-bbcd-08fb9e55e4fb-plugins-config-volume\") pod \"payload-processing-external-model-99c554458-p54jm\" (UID: \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\") " pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" Oct 30 12:50:40.588556 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.588526 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlck2\" (UniqueName: \"kubernetes.io/projected/15c51358-a619-47d6-bbcd-08fb9e55e4fb-kube-api-access-jlck2\") pod \"payload-processing-external-model-99c554458-p54jm\" (UID: \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\") " pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" Oct 30 12:50:40.589055 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.589007 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15c51358-a619-47d6-bbcd-08fb9e55e4fb-routing-overlay-volume\") pod \"payload-processing-external-model-99c554458-p54jm\" (UID: \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\") " pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" Oct 30 12:50:40.589546 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.589493 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15c51358-a619-47d6-bbcd-08fb9e55e4fb-provider-credentials\") pod \"payload-processing-external-model-99c554458-p54jm\" (UID: \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\") " pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" Oct 30 12:50:40.691678 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.691636 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c51358-a619-47d6-bbcd-08fb9e55e4fb-plugins-config-volume\") pod \"payload-processing-external-model-99c554458-p54jm\" (UID: \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\") " pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" Oct 30 12:50:40.691911 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.691699 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlck2\" (UniqueName: \"kubernetes.io/projected/15c51358-a619-47d6-bbcd-08fb9e55e4fb-kube-api-access-jlck2\") pod \"payload-processing-external-model-99c554458-p54jm\" (UID: \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\") " pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" Oct 30 12:50:40.691911 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.691723 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15c51358-a619-47d6-bbcd-08fb9e55e4fb-routing-overlay-volume\") pod \"payload-processing-external-model-99c554458-p54jm\" (UID: \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\") " pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" Oct 30 12:50:40.691911 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.691802 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15c51358-a619-47d6-bbcd-08fb9e55e4fb-provider-credentials\") pod \"payload-processing-external-model-99c554458-p54jm\" (UID: \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\") " pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" Oct 30 12:50:40.692483 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.692455 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15c51358-a619-47d6-bbcd-08fb9e55e4fb-routing-overlay-volume\") pod \"payload-processing-external-model-99c554458-p54jm\" (UID: \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\") " pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" Oct 30 12:50:40.692582 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.692513 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c51358-a619-47d6-bbcd-08fb9e55e4fb-plugins-config-volume\") pod \"payload-processing-external-model-99c554458-p54jm\" (UID: \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\") " pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" Oct 30 12:50:40.694415 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.694394 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15c51358-a619-47d6-bbcd-08fb9e55e4fb-provider-credentials\") pod \"payload-processing-external-model-99c554458-p54jm\" (UID: \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\") " pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" Oct 30 12:50:40.716651 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.716614 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlck2\" (UniqueName: \"kubernetes.io/projected/15c51358-a619-47d6-bbcd-08fb9e55e4fb-kube-api-access-jlck2\") pod \"payload-processing-external-model-99c554458-p54jm\" (UID: \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\") " pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" Oct 30 12:50:40.799186 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:40.799045 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" Oct 30 12:50:41.113527 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:50:41.113489 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c51358_a619_47d6_bbcd_08fb9e55e4fb.slice/crio-4f94e5abfb0977e9844e53234b4434a7b9b46565e4331a90ff4b2c144079ef96 WatchSource:0}: Error finding container 4f94e5abfb0977e9844e53234b4434a7b9b46565e4331a90ff4b2c144079ef96: Status 404 returned error can't find the container with id 4f94e5abfb0977e9844e53234b4434a7b9b46565e4331a90ff4b2c144079ef96 Oct 30 12:50:41.117056 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:41.117030 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-99c554458-p54jm"] Oct 30 12:50:41.465925 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:41.465879 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" event={"ID":"15c51358-a619-47d6-bbcd-08fb9e55e4fb","Type":"ContainerStarted","Data":"4c27a91a0007922d7b3442e2161ed5d5db13279e8952cc66d129c9c2aabaaa36"} Oct 30 12:50:41.466205 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:41.465941 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" event={"ID":"15c51358-a619-47d6-bbcd-08fb9e55e4fb","Type":"ContainerStarted","Data":"4f94e5abfb0977e9844e53234b4434a7b9b46565e4331a90ff4b2c144079ef96"} Oct 30 12:50:41.466205 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:41.465962 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" Oct 30 12:50:41.512010 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:41.511954 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" podStartSLOduration=1.511939699 podStartE2EDuration="1.511939699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:41.509636649 +0000 UTC m=+3392.543775046" watchObservedRunningTime="2026-10-30 12:50:41.511939699 +0000 UTC m=+3392.546078095" Oct 30 12:50:52.477502 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:52.477463 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" Oct 30 12:50:52.581955 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:52.581633 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp"] Oct 30 12:50:53.614829 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:50:53.614784 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:50:53.652156 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:53.652118 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2"] Oct 30 12:50:57.583244 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:57.583186 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" podUID="409c8c14-07b1-42da-a6a5-7e4216abcd20" containerName="payload-processing" containerID="cri-o://0f78c4123e904f5bdefbcb21fe96c681738ba0f4849c3ea14389baa299d68b53" gracePeriod=25 Oct 30 12:50:57.872985 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:57.872953 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" Oct 30 12:50:57.991330 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:57.991293 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/409c8c14-07b1-42da-a6a5-7e4216abcd20-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/409c8c14-07b1-42da-a6a5-7e4216abcd20-routing-overlay-volume\") pod \"409c8c14-07b1-42da-a6a5-7e4216abcd20\" (UID: \"409c8c14-07b1-42da-a6a5-7e4216abcd20\") " Oct 30 12:50:57.991553 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:57.991534 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409c8c14-07b1-42da-a6a5-7e4216abcd20-provider-credentials\" (UniqueName: \"kubernetes.io/projected/409c8c14-07b1-42da-a6a5-7e4216abcd20-provider-credentials\") pod \"409c8c14-07b1-42da-a6a5-7e4216abcd20\" (UID: \"409c8c14-07b1-42da-a6a5-7e4216abcd20\") " Oct 30 12:50:57.991654 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:57.991636 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/409c8c14-07b1-42da-a6a5-7e4216abcd20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409c8c14-07b1-42da-a6a5-7e4216abcd20-plugins-config-volume\") pod \"409c8c14-07b1-42da-a6a5-7e4216abcd20\" (UID: \"409c8c14-07b1-42da-a6a5-7e4216abcd20\") " Oct 30 12:50:57.991721 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:57.991702 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409c8c14-07b1-42da-a6a5-7e4216abcd20-kube-api-access-sxgnw\" (UniqueName: \"kubernetes.io/projected/409c8c14-07b1-42da-a6a5-7e4216abcd20-kube-api-access-sxgnw\") pod \"409c8c14-07b1-42da-a6a5-7e4216abcd20\" (UID: \"409c8c14-07b1-42da-a6a5-7e4216abcd20\") " Oct 30 12:50:57.991776 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:57.991717 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409c8c14-07b1-42da-a6a5-7e4216abcd20-routing-overlay-volume" pod "409c8c14-07b1-42da-a6a5-7e4216abcd20" (UID: "409c8c14-07b1-42da-a6a5-7e4216abcd20"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:57.992011 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:57.991989 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409c8c14-07b1-42da-a6a5-7e4216abcd20-plugins-config-volume" pod "409c8c14-07b1-42da-a6a5-7e4216abcd20" (UID: "409c8c14-07b1-42da-a6a5-7e4216abcd20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:57.992288 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:57.992269 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/409c8c14-07b1-42da-a6a5-7e4216abcd20-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:50:57.992379 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:57.992289 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409c8c14-07b1-42da-a6a5-7e4216abcd20-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:50:57.993973 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:57.993945 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409c8c14-07b1-42da-a6a5-7e4216abcd20-provider-credentials" pod "409c8c14-07b1-42da-a6a5-7e4216abcd20" (UID: "409c8c14-07b1-42da-a6a5-7e4216abcd20"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:57.994134 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:57.994002 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409c8c14-07b1-42da-a6a5-7e4216abcd20-kube-api-access-sxgnw" pod "409c8c14-07b1-42da-a6a5-7e4216abcd20" (UID: "409c8c14-07b1-42da-a6a5-7e4216abcd20"). InnerVolumeSpecName "kube-api-access-sxgnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:58.094147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:58.094005 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/409c8c14-07b1-42da-a6a5-7e4216abcd20-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:50:58.094147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:58.094046 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxgnw\" (UniqueName: \"kubernetes.io/projected/409c8c14-07b1-42da-a6a5-7e4216abcd20-kube-api-access-sxgnw\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:50:58.653335 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:58.653291 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" podUID="e9391811-2e96-4b3d-879a-c631c4896ed8" containerName="payload-processing" containerID="cri-o://0589e22b0e8e15d80e07d663722dea7080ce06a392d7f711a6238636b6ca67c0" gracePeriod=25 Oct 30 12:50:58.670350 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:58.670313 4468 generic.go:356] "Generic (PLEG): container finished" podID="409c8c14-07b1-42da-a6a5-7e4216abcd20" containerID="0f78c4123e904f5bdefbcb21fe96c681738ba0f4849c3ea14389baa299d68b53" exitCode=0 Oct 30 12:50:58.670527 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:58.670357 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" Oct 30 12:50:58.670527 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:58.670390 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" event={"ID":"409c8c14-07b1-42da-a6a5-7e4216abcd20","Type":"ContainerDied","Data":"0f78c4123e904f5bdefbcb21fe96c681738ba0f4849c3ea14389baa299d68b53"} Oct 30 12:50:58.670527 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:58.670457 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp" event={"ID":"409c8c14-07b1-42da-a6a5-7e4216abcd20","Type":"ContainerDied","Data":"83012bd8d130edb98d58b6a84899003570e103619238fd425c1bbe3c9a2e3025"} Oct 30 12:50:58.670527 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:58.670487 4468 scope.go:122] "RemoveContainer" containerID="0f78c4123e904f5bdefbcb21fe96c681738ba0f4849c3ea14389baa299d68b53" Oct 30 12:50:58.767689 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:58.767660 4468 scope.go:122] "RemoveContainer" containerID="0f78c4123e904f5bdefbcb21fe96c681738ba0f4849c3ea14389baa299d68b53" Oct 30 12:50:58.768042 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:50:58.768018 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f78c4123e904f5bdefbcb21fe96c681738ba0f4849c3ea14389baa299d68b53\": container with ID starting with 0f78c4123e904f5bdefbcb21fe96c681738ba0f4849c3ea14389baa299d68b53 not found: ID does not exist" containerID="0f78c4123e904f5bdefbcb21fe96c681738ba0f4849c3ea14389baa299d68b53" Oct 30 12:50:58.768177 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:58.768054 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f78c4123e904f5bdefbcb21fe96c681738ba0f4849c3ea14389baa299d68b53"} err="failed to get container status \"0f78c4123e904f5bdefbcb21fe96c681738ba0f4849c3ea14389baa299d68b53\": rpc error: code = NotFound desc = could not find container \"0f78c4123e904f5bdefbcb21fe96c681738ba0f4849c3ea14389baa299d68b53\": container with ID starting with 0f78c4123e904f5bdefbcb21fe96c681738ba0f4849c3ea14389baa299d68b53 not found: ID does not exist" Oct 30 12:50:58.772913 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:58.772869 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp"] Oct 30 12:50:58.785886 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:58.785851 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-869554dfbc-tvkrp"] Oct 30 12:50:58.964667 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:58.964627 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" Oct 30 12:50:59.096691 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.096650 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="409c8c14-07b1-42da-a6a5-7e4216abcd20" path="/var/lib/kubelet/pods/409c8c14-07b1-42da-a6a5-7e4216abcd20/volumes" Oct 30 12:50:59.112356 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.112322 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9391811-2e96-4b3d-879a-c631c4896ed8-kube-api-access-sjfcs\" (UniqueName: \"kubernetes.io/projected/e9391811-2e96-4b3d-879a-c631c4896ed8-kube-api-access-sjfcs\") pod \"e9391811-2e96-4b3d-879a-c631c4896ed8\" (UID: \"e9391811-2e96-4b3d-879a-c631c4896ed8\") " Oct 30 12:50:59.112518 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.112405 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9391811-2e96-4b3d-879a-c631c4896ed8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9391811-2e96-4b3d-879a-c631c4896ed8-routing-overlay-volume\") pod \"e9391811-2e96-4b3d-879a-c631c4896ed8\" (UID: \"e9391811-2e96-4b3d-879a-c631c4896ed8\") " Oct 30 12:50:59.112590 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.112522 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9391811-2e96-4b3d-879a-c631c4896ed8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9391811-2e96-4b3d-879a-c631c4896ed8-plugins-config-volume\") pod \"e9391811-2e96-4b3d-879a-c631c4896ed8\" (UID: \"e9391811-2e96-4b3d-879a-c631c4896ed8\") " Oct 30 12:50:59.112652 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.112609 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9391811-2e96-4b3d-879a-c631c4896ed8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9391811-2e96-4b3d-879a-c631c4896ed8-provider-credentials\") pod \"e9391811-2e96-4b3d-879a-c631c4896ed8\" (UID: \"e9391811-2e96-4b3d-879a-c631c4896ed8\") " Oct 30 12:50:59.112844 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.112810 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9391811-2e96-4b3d-879a-c631c4896ed8-routing-overlay-volume" pod "e9391811-2e96-4b3d-879a-c631c4896ed8" (UID: "e9391811-2e96-4b3d-879a-c631c4896ed8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:59.112971 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.112916 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9391811-2e96-4b3d-879a-c631c4896ed8-plugins-config-volume" pod "e9391811-2e96-4b3d-879a-c631c4896ed8" (UID: "e9391811-2e96-4b3d-879a-c631c4896ed8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:59.113168 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.113151 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9391811-2e96-4b3d-879a-c631c4896ed8-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:50:59.113239 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.113170 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9391811-2e96-4b3d-879a-c631c4896ed8-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:50:59.114669 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.114649 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9391811-2e96-4b3d-879a-c631c4896ed8-kube-api-access-sjfcs" pod "e9391811-2e96-4b3d-879a-c631c4896ed8" (UID: "e9391811-2e96-4b3d-879a-c631c4896ed8"). InnerVolumeSpecName "kube-api-access-sjfcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:59.114742 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.114700 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9391811-2e96-4b3d-879a-c631c4896ed8-provider-credentials" pod "e9391811-2e96-4b3d-879a-c631c4896ed8" (UID: "e9391811-2e96-4b3d-879a-c631c4896ed8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:59.215382 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.215273 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjfcs\" (UniqueName: \"kubernetes.io/projected/e9391811-2e96-4b3d-879a-c631c4896ed8-kube-api-access-sjfcs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:50:59.215382 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.215327 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9391811-2e96-4b3d-879a-c631c4896ed8-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:50:59.680342 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.680308 4468 generic.go:356] "Generic (PLEG): container finished" podID="e9391811-2e96-4b3d-879a-c631c4896ed8" containerID="0589e22b0e8e15d80e07d663722dea7080ce06a392d7f711a6238636b6ca67c0" exitCode=0 Oct 30 12:50:59.680829 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.680354 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" Oct 30 12:50:59.680829 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.680364 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" event={"ID":"e9391811-2e96-4b3d-879a-c631c4896ed8","Type":"ContainerDied","Data":"0589e22b0e8e15d80e07d663722dea7080ce06a392d7f711a6238636b6ca67c0"} Oct 30 12:50:59.680829 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.680436 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2" event={"ID":"e9391811-2e96-4b3d-879a-c631c4896ed8","Type":"ContainerDied","Data":"eb754ef90f20cd83d371303af6a7bf9aec70278090e3cb75987ff703f38a1d14"} Oct 30 12:50:59.680829 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.680469 4468 scope.go:122] "RemoveContainer" containerID="0589e22b0e8e15d80e07d663722dea7080ce06a392d7f711a6238636b6ca67c0" Oct 30 12:50:59.692123 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.692100 4468 scope.go:122] "RemoveContainer" containerID="0589e22b0e8e15d80e07d663722dea7080ce06a392d7f711a6238636b6ca67c0" Oct 30 12:50:59.692473 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:50:59.692447 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0589e22b0e8e15d80e07d663722dea7080ce06a392d7f711a6238636b6ca67c0\": container with ID starting with 0589e22b0e8e15d80e07d663722dea7080ce06a392d7f711a6238636b6ca67c0 not found: ID does not exist" containerID="0589e22b0e8e15d80e07d663722dea7080ce06a392d7f711a6238636b6ca67c0" Oct 30 12:50:59.692577 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.692481 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0589e22b0e8e15d80e07d663722dea7080ce06a392d7f711a6238636b6ca67c0"} err="failed to get container status \"0589e22b0e8e15d80e07d663722dea7080ce06a392d7f711a6238636b6ca67c0\": rpc error: code = NotFound desc = could not find container \"0589e22b0e8e15d80e07d663722dea7080ce06a392d7f711a6238636b6ca67c0\": container with ID starting with 0589e22b0e8e15d80e07d663722dea7080ce06a392d7f711a6238636b6ca67c0 not found: ID does not exist" Oct 30 12:50:59.740510 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.740478 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2"] Oct 30 12:50:59.753796 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:50:59.753764 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-9ffdkhgn2"] Oct 30 12:51:01.092134 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:01.092102 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9391811-2e96-4b3d-879a-c631c4896ed8" path="/var/lib/kubelet/pods/e9391811-2e96-4b3d-879a-c631c4896ed8/volumes" Oct 30 12:51:17.589123 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.589087 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf"] Oct 30 12:51:17.590078 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.590044 4468 cpu_manager.go:414] "containerMap: removing container" podUID="409c8c14-07b1-42da-a6a5-7e4216abcd20" containerName="payload-processing" Oct 30 12:51:17.590078 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.590079 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409c8c14-07b1-42da-a6a5-7e4216abcd20" containerName="payload-processing" Oct 30 12:51:17.590208 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.590132 4468 cpu_manager.go:414] "containerMap: removing container" podUID="e9391811-2e96-4b3d-879a-c631c4896ed8" containerName="payload-processing" Oct 30 12:51:17.590208 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.590139 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9391811-2e96-4b3d-879a-c631c4896ed8" containerName="payload-processing" Oct 30 12:51:17.590276 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.590260 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="409c8c14-07b1-42da-a6a5-7e4216abcd20" containerName="payload-processing" Oct 30 12:51:17.590314 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.590277 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9391811-2e96-4b3d-879a-c631c4896ed8" containerName="payload-processing" Oct 30 12:51:17.594626 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.594600 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" Oct 30 12:51:17.613147 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.613118 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf"] Oct 30 12:51:17.618982 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.618935 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0f7d14\"/\"agc-cred-ee2e66e280d5\"" Oct 30 12:51:17.618982 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.618955 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f7d14\"/\"routing-overlay\"" Oct 30 12:51:17.618982 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.618955 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0f7d14\"/\"payload-processing-external-model-e2e--d59b9242-dockercfg-m4wpj\"" Oct 30 12:51:17.618982 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.618935 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f7d14\"/\"openshift-service-ca.crt\"" Oct 30 12:51:17.624748 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.624722 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f7d14\"/\"kube-root-ca.crt\"" Oct 30 12:51:17.624871 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.624722 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f7d14\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0f7d14\"" Oct 30 12:51:17.704409 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.704374 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d47a9c7-345b-4111-bec3-a9a5801ba911-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf\" (UID: \"9d47a9c7-345b-4111-bec3-a9a5801ba911\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" Oct 30 12:51:17.704596 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.704450 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbfzk\" (UniqueName: \"kubernetes.io/projected/9d47a9c7-345b-4111-bec3-a9a5801ba911-kube-api-access-nbfzk\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf\" (UID: \"9d47a9c7-345b-4111-bec3-a9a5801ba911\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" Oct 30 12:51:17.704654 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.704578 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d47a9c7-345b-4111-bec3-a9a5801ba911-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf\" (UID: \"9d47a9c7-345b-4111-bec3-a9a5801ba911\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" Oct 30 12:51:17.704744 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.704726 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d47a9c7-345b-4111-bec3-a9a5801ba911-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf\" (UID: \"9d47a9c7-345b-4111-bec3-a9a5801ba911\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" Oct 30 12:51:17.808260 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.808182 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d47a9c7-345b-4111-bec3-a9a5801ba911-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf\" (UID: \"9d47a9c7-345b-4111-bec3-a9a5801ba911\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" Oct 30 12:51:17.816099 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.809604 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d47a9c7-345b-4111-bec3-a9a5801ba911-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf\" (UID: \"9d47a9c7-345b-4111-bec3-a9a5801ba911\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" Oct 30 12:51:17.817480 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.817447 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbfzk\" (UniqueName: \"kubernetes.io/projected/9d47a9c7-345b-4111-bec3-a9a5801ba911-kube-api-access-nbfzk\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf\" (UID: \"9d47a9c7-345b-4111-bec3-a9a5801ba911\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" Oct 30 12:51:17.818167 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.818129 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d47a9c7-345b-4111-bec3-a9a5801ba911-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf\" (UID: \"9d47a9c7-345b-4111-bec3-a9a5801ba911\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" Oct 30 12:51:17.818751 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.809316 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d47a9c7-345b-4111-bec3-a9a5801ba911-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf\" (UID: \"9d47a9c7-345b-4111-bec3-a9a5801ba911\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" Oct 30 12:51:17.826061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.826024 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d47a9c7-345b-4111-bec3-a9a5801ba911-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf\" (UID: \"9d47a9c7-345b-4111-bec3-a9a5801ba911\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" Oct 30 12:51:17.833584 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.833543 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d47a9c7-345b-4111-bec3-a9a5801ba911-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf\" (UID: \"9d47a9c7-345b-4111-bec3-a9a5801ba911\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" Oct 30 12:51:17.843013 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.842938 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbfzk\" (UniqueName: \"kubernetes.io/projected/9d47a9c7-345b-4111-bec3-a9a5801ba911-kube-api-access-nbfzk\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf\" (UID: \"9d47a9c7-345b-4111-bec3-a9a5801ba911\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" Oct 30 12:51:17.912166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:17.912131 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" Oct 30 12:51:18.241551 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:51:18.241517 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d47a9c7_345b_4111_bec3_a9a5801ba911.slice/crio-e465d3f37e79b4ad2aa700424d3a1ea79a0f17373736495d3c41526a4c4e65b1 WatchSource:0}: Error finding container e465d3f37e79b4ad2aa700424d3a1ea79a0f17373736495d3c41526a4c4e65b1: Status 404 returned error can't find the container with id e465d3f37e79b4ad2aa700424d3a1ea79a0f17373736495d3c41526a4c4e65b1 Oct 30 12:51:18.245472 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.245450 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf"] Oct 30 12:51:18.533388 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.533302 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn"] Oct 30 12:51:18.538136 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.538108 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" Oct 30 12:51:18.557142 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.556240 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a786af\"" Oct 30 12:51:18.557585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.557175 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn"] Oct 30 12:51:18.567362 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.565154 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"routing-overlay\"" Oct 30 12:51:18.567362 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.565175 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"payload-processing-external-model-e2e--3f99fe6c-dockercfg-rcn55\"" Oct 30 12:51:18.567362 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.565166 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"openshift-service-ca.crt\"" Oct 30 12:51:18.567362 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.565170 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"kube-root-ca.crt\"" Oct 30 12:51:18.567362 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.565221 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a786af\"/\"agc-cred-e690b3435b89\"" Oct 30 12:51:18.633935 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.633889 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krqv2\" (UniqueName: \"kubernetes.io/projected/8f86a91b-53e8-4cee-95e8-086df48499e2-kube-api-access-krqv2\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn\" (UID: \"8f86a91b-53e8-4cee-95e8-086df48499e2\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" Oct 30 12:51:18.634411 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.634156 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f86a91b-53e8-4cee-95e8-086df48499e2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn\" (UID: \"8f86a91b-53e8-4cee-95e8-086df48499e2\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" Oct 30 12:51:18.634411 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.634202 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f86a91b-53e8-4cee-95e8-086df48499e2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn\" (UID: \"8f86a91b-53e8-4cee-95e8-086df48499e2\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" Oct 30 12:51:18.634411 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.634312 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f86a91b-53e8-4cee-95e8-086df48499e2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn\" (UID: \"8f86a91b-53e8-4cee-95e8-086df48499e2\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" Oct 30 12:51:18.737337 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.737286 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f86a91b-53e8-4cee-95e8-086df48499e2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn\" (UID: \"8f86a91b-53e8-4cee-95e8-086df48499e2\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" Oct 30 12:51:18.738713 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.738686 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krqv2\" (UniqueName: \"kubernetes.io/projected/8f86a91b-53e8-4cee-95e8-086df48499e2-kube-api-access-krqv2\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn\" (UID: \"8f86a91b-53e8-4cee-95e8-086df48499e2\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" Oct 30 12:51:18.739442 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.739412 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f86a91b-53e8-4cee-95e8-086df48499e2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn\" (UID: \"8f86a91b-53e8-4cee-95e8-086df48499e2\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" Oct 30 12:51:18.739568 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.739461 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f86a91b-53e8-4cee-95e8-086df48499e2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn\" (UID: \"8f86a91b-53e8-4cee-95e8-086df48499e2\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" Oct 30 12:51:18.740062 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.740037 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f86a91b-53e8-4cee-95e8-086df48499e2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn\" (UID: \"8f86a91b-53e8-4cee-95e8-086df48499e2\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" Oct 30 12:51:18.740062 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.738496 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f86a91b-53e8-4cee-95e8-086df48499e2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn\" (UID: \"8f86a91b-53e8-4cee-95e8-086df48499e2\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" Oct 30 12:51:18.747258 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.744613 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f86a91b-53e8-4cee-95e8-086df48499e2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn\" (UID: \"8f86a91b-53e8-4cee-95e8-086df48499e2\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" Oct 30 12:51:18.778902 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.778867 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krqv2\" (UniqueName: \"kubernetes.io/projected/8f86a91b-53e8-4cee-95e8-086df48499e2-kube-api-access-krqv2\") pod \"payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn\" (UID: \"8f86a91b-53e8-4cee-95e8-086df48499e2\") " pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" Oct 30 12:51:18.865194 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.863885 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" Oct 30 12:51:18.885085 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.885036 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" event={"ID":"9d47a9c7-345b-4111-bec3-a9a5801ba911","Type":"ContainerStarted","Data":"a819be975bd1b3e102522fb0baa50319723ad7e8870415b495a66ea3c62700d0"} Oct 30 12:51:18.885280 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.885153 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" Oct 30 12:51:18.885280 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.885182 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" event={"ID":"9d47a9c7-345b-4111-bec3-a9a5801ba911","Type":"ContainerStarted","Data":"e465d3f37e79b4ad2aa700424d3a1ea79a0f17373736495d3c41526a4c4e65b1"} Oct 30 12:51:18.932057 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:18.931998 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" podStartSLOduration=1.931982962 podStartE2EDuration="1.931982962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:18.930913315 +0000 UTC m=+3429.965051712" watchObservedRunningTime="2026-10-30 12:51:18.931982962 +0000 UTC m=+3429.966121357" Oct 30 12:51:19.189155 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:51:19.189118 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f86a91b_53e8_4cee_95e8_086df48499e2.slice/crio-cbe1b08c869ffd9cb112b9e3ccb1ff37d13e1dee67f009f5877eeff5b95e29ee WatchSource:0}: Error finding container cbe1b08c869ffd9cb112b9e3ccb1ff37d13e1dee67f009f5877eeff5b95e29ee: Status 404 returned error can't find the container with id cbe1b08c869ffd9cb112b9e3ccb1ff37d13e1dee67f009f5877eeff5b95e29ee Oct 30 12:51:19.193190 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:19.193168 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn"] Oct 30 12:51:19.894693 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:19.894650 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" event={"ID":"8f86a91b-53e8-4cee-95e8-086df48499e2","Type":"ContainerStarted","Data":"287e32dc52ac7b454f1d69430246873b4cb6679baf3b3c325a45cce4f7660f5a"} Oct 30 12:51:19.895193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:19.894848 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" Oct 30 12:51:19.895193 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:19.894887 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" event={"ID":"8f86a91b-53e8-4cee-95e8-086df48499e2","Type":"ContainerStarted","Data":"cbe1b08c869ffd9cb112b9e3ccb1ff37d13e1dee67f009f5877eeff5b95e29ee"} Oct 30 12:51:19.942135 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:19.942045 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" podStartSLOduration=1.942029943 podStartE2EDuration="1.942029943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:19.938225037 +0000 UTC m=+3430.972363432" watchObservedRunningTime="2026-10-30 12:51:19.942029943 +0000 UTC m=+3430.976168340" Oct 30 12:51:23.601646 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:51:23.601602 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:51:29.898705 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:29.898666 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" Oct 30 12:51:30.906276 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:51:30.906245 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" Oct 30 12:51:53.602684 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:51:53.602598 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:52:23.600874 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:52:23.600837 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:52:53.601860 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:52:53.601816 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:53:23.602229 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:53:23.602142 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:53:53.602379 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:53:53.602333 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:53:57.598367 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:53:57.598323 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-99c554458-p54jm"] Oct 30 12:54:02.599765 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:02.599721 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" podUID="15c51358-a619-47d6-bbcd-08fb9e55e4fb" containerName="payload-processing" containerID="cri-o://4c27a91a0007922d7b3442e2161ed5d5db13279e8952cc66d129c9c2aabaaa36" gracePeriod=25 Oct 30 12:54:02.909954 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:02.909914 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" Oct 30 12:54:02.938012 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:02.937978 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c51358-a619-47d6-bbcd-08fb9e55e4fb-kube-api-access-jlck2\" (UniqueName: \"kubernetes.io/projected/15c51358-a619-47d6-bbcd-08fb9e55e4fb-kube-api-access-jlck2\") pod \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\" (UID: \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\") " Oct 30 12:54:02.938212 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:02.938133 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c51358-a619-47d6-bbcd-08fb9e55e4fb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/15c51358-a619-47d6-bbcd-08fb9e55e4fb-provider-credentials\") pod \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\" (UID: \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\") " Oct 30 12:54:02.938267 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:02.938224 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c51358-a619-47d6-bbcd-08fb9e55e4fb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15c51358-a619-47d6-bbcd-08fb9e55e4fb-routing-overlay-volume\") pod \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\" (UID: \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\") " Oct 30 12:54:02.938267 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:02.938247 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c51358-a619-47d6-bbcd-08fb9e55e4fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c51358-a619-47d6-bbcd-08fb9e55e4fb-plugins-config-volume\") pod \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\" (UID: \"15c51358-a619-47d6-bbcd-08fb9e55e4fb\") " Oct 30 12:54:02.938677 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:02.938649 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c51358-a619-47d6-bbcd-08fb9e55e4fb-routing-overlay-volume" pod "15c51358-a619-47d6-bbcd-08fb9e55e4fb" (UID: "15c51358-a619-47d6-bbcd-08fb9e55e4fb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:54:02.938776 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:02.938665 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c51358-a619-47d6-bbcd-08fb9e55e4fb-plugins-config-volume" pod "15c51358-a619-47d6-bbcd-08fb9e55e4fb" (UID: "15c51358-a619-47d6-bbcd-08fb9e55e4fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:54:02.938852 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:02.938836 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15c51358-a619-47d6-bbcd-08fb9e55e4fb-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:54:02.938913 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:02.938855 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c51358-a619-47d6-bbcd-08fb9e55e4fb-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:54:02.940294 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:02.940266 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c51358-a619-47d6-bbcd-08fb9e55e4fb-kube-api-access-jlck2" pod "15c51358-a619-47d6-bbcd-08fb9e55e4fb" (UID: "15c51358-a619-47d6-bbcd-08fb9e55e4fb"). InnerVolumeSpecName "kube-api-access-jlck2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:02.940386 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:02.940336 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c51358-a619-47d6-bbcd-08fb9e55e4fb-provider-credentials" pod "15c51358-a619-47d6-bbcd-08fb9e55e4fb" (UID: "15c51358-a619-47d6-bbcd-08fb9e55e4fb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:03.040174 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:03.040126 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15c51358-a619-47d6-bbcd-08fb9e55e4fb-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:54:03.040174 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:03.040171 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlck2\" (UniqueName: \"kubernetes.io/projected/15c51358-a619-47d6-bbcd-08fb9e55e4fb-kube-api-access-jlck2\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:54:03.292253 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:03.292212 4468 generic.go:356] "Generic (PLEG): container finished" podID="15c51358-a619-47d6-bbcd-08fb9e55e4fb" containerID="4c27a91a0007922d7b3442e2161ed5d5db13279e8952cc66d129c9c2aabaaa36" exitCode=0 Oct 30 12:54:03.292469 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:03.292266 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" Oct 30 12:54:03.292469 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:03.292300 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" event={"ID":"15c51358-a619-47d6-bbcd-08fb9e55e4fb","Type":"ContainerDied","Data":"4c27a91a0007922d7b3442e2161ed5d5db13279e8952cc66d129c9c2aabaaa36"} Oct 30 12:54:03.292469 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:03.292359 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-99c554458-p54jm" event={"ID":"15c51358-a619-47d6-bbcd-08fb9e55e4fb","Type":"ContainerDied","Data":"4f94e5abfb0977e9844e53234b4434a7b9b46565e4331a90ff4b2c144079ef96"} Oct 30 12:54:03.292469 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:03.292381 4468 scope.go:122] "RemoveContainer" containerID="4c27a91a0007922d7b3442e2161ed5d5db13279e8952cc66d129c9c2aabaaa36" Oct 30 12:54:03.303230 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:03.303209 4468 scope.go:122] "RemoveContainer" containerID="4c27a91a0007922d7b3442e2161ed5d5db13279e8952cc66d129c9c2aabaaa36" Oct 30 12:54:03.303597 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:54:03.303576 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c27a91a0007922d7b3442e2161ed5d5db13279e8952cc66d129c9c2aabaaa36\": container with ID starting with 4c27a91a0007922d7b3442e2161ed5d5db13279e8952cc66d129c9c2aabaaa36 not found: ID does not exist" containerID="4c27a91a0007922d7b3442e2161ed5d5db13279e8952cc66d129c9c2aabaaa36" Oct 30 12:54:03.303660 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:03.303610 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c27a91a0007922d7b3442e2161ed5d5db13279e8952cc66d129c9c2aabaaa36"} err="failed to get container status \"4c27a91a0007922d7b3442e2161ed5d5db13279e8952cc66d129c9c2aabaaa36\": rpc error: code = NotFound desc = could not find container \"4c27a91a0007922d7b3442e2161ed5d5db13279e8952cc66d129c9c2aabaaa36\": container with ID starting with 4c27a91a0007922d7b3442e2161ed5d5db13279e8952cc66d129c9c2aabaaa36 not found: ID does not exist" Oct 30 12:54:03.353144 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:03.353107 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-99c554458-p54jm"] Oct 30 12:54:03.366706 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:03.366668 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-99c554458-p54jm"] Oct 30 12:54:05.092794 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:05.092758 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15c51358-a619-47d6-bbcd-08fb9e55e4fb" path="/var/lib/kubelet/pods/15c51358-a619-47d6-bbcd-08fb9e55e4fb/volumes" Oct 30 12:54:08.439656 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.439619 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv"] Oct 30 12:54:08.440539 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.440520 4468 cpu_manager.go:414] "containerMap: removing container" podUID="15c51358-a619-47d6-bbcd-08fb9e55e4fb" containerName="payload-processing" Oct 30 12:54:08.440593 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.440540 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c51358-a619-47d6-bbcd-08fb9e55e4fb" containerName="payload-processing" Oct 30 12:54:08.440671 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.440661 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c51358-a619-47d6-bbcd-08fb9e55e4fb" containerName="payload-processing" Oct 30 12:54:08.446424 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.446399 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" Oct 30 12:54:08.463201 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.463173 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"routing-overlay\"" Oct 30 12:54:08.464383 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.464360 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv"] Oct 30 12:54:08.474125 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.471441 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"openshift-service-ca.crt\"" Oct 30 12:54:08.474125 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.471442 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"kube-root-ca.crt\"" Oct 30 12:54:08.474125 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.471476 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"agc-cred-bbd00a81253a\"" Oct 30 12:54:08.477045 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.477014 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2545c7\"" Oct 30 12:54:08.477223 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.477200 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"payload-processing-external-model-e2e--03fcd75d-dockercfg-5q2hg\"" Oct 30 12:54:08.569327 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.569283 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01456588-661b-424b-a6b0-3ebcca0cc233-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv\" (UID: \"01456588-661b-424b-a6b0-3ebcca0cc233\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" Oct 30 12:54:08.569530 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.569409 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01456588-661b-424b-a6b0-3ebcca0cc233-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv\" (UID: \"01456588-661b-424b-a6b0-3ebcca0cc233\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" Oct 30 12:54:08.569530 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.569472 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01456588-661b-424b-a6b0-3ebcca0cc233-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv\" (UID: \"01456588-661b-424b-a6b0-3ebcca0cc233\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" Oct 30 12:54:08.569655 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.569634 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbbhc\" (UniqueName: \"kubernetes.io/projected/01456588-661b-424b-a6b0-3ebcca0cc233-kube-api-access-tbbhc\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv\" (UID: \"01456588-661b-424b-a6b0-3ebcca0cc233\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" Oct 30 12:54:08.671652 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.671614 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbbhc\" (UniqueName: \"kubernetes.io/projected/01456588-661b-424b-a6b0-3ebcca0cc233-kube-api-access-tbbhc\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv\" (UID: \"01456588-661b-424b-a6b0-3ebcca0cc233\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" Oct 30 12:54:08.671880 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.671763 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01456588-661b-424b-a6b0-3ebcca0cc233-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv\" (UID: \"01456588-661b-424b-a6b0-3ebcca0cc233\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" Oct 30 12:54:08.671880 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.671802 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01456588-661b-424b-a6b0-3ebcca0cc233-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv\" (UID: \"01456588-661b-424b-a6b0-3ebcca0cc233\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" Oct 30 12:54:08.671880 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.671862 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01456588-661b-424b-a6b0-3ebcca0cc233-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv\" (UID: \"01456588-661b-424b-a6b0-3ebcca0cc233\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" Oct 30 12:54:08.672697 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.672675 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01456588-661b-424b-a6b0-3ebcca0cc233-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv\" (UID: \"01456588-661b-424b-a6b0-3ebcca0cc233\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" Oct 30 12:54:08.672821 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.672724 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01456588-661b-424b-a6b0-3ebcca0cc233-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv\" (UID: \"01456588-661b-424b-a6b0-3ebcca0cc233\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" Oct 30 12:54:08.674398 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.674376 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01456588-661b-424b-a6b0-3ebcca0cc233-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv\" (UID: \"01456588-661b-424b-a6b0-3ebcca0cc233\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" Oct 30 12:54:08.697205 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.697123 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbbhc\" (UniqueName: \"kubernetes.io/projected/01456588-661b-424b-a6b0-3ebcca0cc233-kube-api-access-tbbhc\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv\" (UID: \"01456588-661b-424b-a6b0-3ebcca0cc233\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" Oct 30 12:54:08.765624 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:08.765285 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" Oct 30 12:54:09.274154 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:54:09.263211 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01456588_661b_424b_a6b0_3ebcca0cc233.slice/crio-e6db66aa921631922feb90ceb619ee273aa2d626814589300cf7bf3c8e6653b1 WatchSource:0}: Error finding container e6db66aa921631922feb90ceb619ee273aa2d626814589300cf7bf3c8e6653b1: Status 404 returned error can't find the container with id e6db66aa921631922feb90ceb619ee273aa2d626814589300cf7bf3c8e6653b1 Oct 30 12:54:09.286319 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:09.275643 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv"] Oct 30 12:54:09.342359 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:09.342315 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" event={"ID":"01456588-661b-424b-a6b0-3ebcca0cc233","Type":"ContainerStarted","Data":"e6db66aa921631922feb90ceb619ee273aa2d626814589300cf7bf3c8e6653b1"} Oct 30 12:54:10.351845 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:10.351806 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" event={"ID":"01456588-661b-424b-a6b0-3ebcca0cc233","Type":"ContainerStarted","Data":"567ef5a6d6e53cd790a6ecbbf61236717b9f86f1020e2c51840ee733474e2305"} Oct 30 12:54:10.352298 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:10.351910 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" Oct 30 12:54:10.399958 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:10.399900 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" podStartSLOduration=2.399886435 podStartE2EDuration="2.399886435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:10.396809359 +0000 UTC m=+3601.430947755" watchObservedRunningTime="2026-10-30 12:54:10.399886435 +0000 UTC m=+3601.434024831" Oct 30 12:54:21.362813 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:21.362784 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" Oct 30 12:54:23.601317 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:54:23.601273 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:54:32.534174 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:32.534136 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv"] Oct 30 12:54:37.539260 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:37.537191 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" podUID="01456588-661b-424b-a6b0-3ebcca0cc233" containerName="payload-processing" containerID="cri-o://567ef5a6d6e53cd790a6ecbbf61236717b9f86f1020e2c51840ee733474e2305" gracePeriod=25 Oct 30 12:54:38.031700 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.031616 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" Oct 30 12:54:38.131527 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.131483 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01456588-661b-424b-a6b0-3ebcca0cc233-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01456588-661b-424b-a6b0-3ebcca0cc233-plugins-config-volume\") pod \"01456588-661b-424b-a6b0-3ebcca0cc233\" (UID: \"01456588-661b-424b-a6b0-3ebcca0cc233\") " Oct 30 12:54:38.131735 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.131715 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01456588-661b-424b-a6b0-3ebcca0cc233-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01456588-661b-424b-a6b0-3ebcca0cc233-routing-overlay-volume\") pod \"01456588-661b-424b-a6b0-3ebcca0cc233\" (UID: \"01456588-661b-424b-a6b0-3ebcca0cc233\") " Oct 30 12:54:38.131821 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.131771 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01456588-661b-424b-a6b0-3ebcca0cc233-provider-credentials\" (UniqueName: \"kubernetes.io/projected/01456588-661b-424b-a6b0-3ebcca0cc233-provider-credentials\") pod \"01456588-661b-424b-a6b0-3ebcca0cc233\" (UID: \"01456588-661b-424b-a6b0-3ebcca0cc233\") " Oct 30 12:54:38.131907 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.131891 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01456588-661b-424b-a6b0-3ebcca0cc233-kube-api-access-tbbhc\" (UniqueName: \"kubernetes.io/projected/01456588-661b-424b-a6b0-3ebcca0cc233-kube-api-access-tbbhc\") pod \"01456588-661b-424b-a6b0-3ebcca0cc233\" (UID: \"01456588-661b-424b-a6b0-3ebcca0cc233\") " Oct 30 12:54:38.132165 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.132058 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01456588-661b-424b-a6b0-3ebcca0cc233-plugins-config-volume" pod "01456588-661b-424b-a6b0-3ebcca0cc233" (UID: "01456588-661b-424b-a6b0-3ebcca0cc233"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:54:38.132499 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.132471 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01456588-661b-424b-a6b0-3ebcca0cc233-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:54:38.132857 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.132827 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01456588-661b-424b-a6b0-3ebcca0cc233-routing-overlay-volume" pod "01456588-661b-424b-a6b0-3ebcca0cc233" (UID: "01456588-661b-424b-a6b0-3ebcca0cc233"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:54:38.134661 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.134620 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01456588-661b-424b-a6b0-3ebcca0cc233-kube-api-access-tbbhc" pod "01456588-661b-424b-a6b0-3ebcca0cc233" (UID: "01456588-661b-424b-a6b0-3ebcca0cc233"). InnerVolumeSpecName "kube-api-access-tbbhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:38.135206 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.135184 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01456588-661b-424b-a6b0-3ebcca0cc233-provider-credentials" pod "01456588-661b-424b-a6b0-3ebcca0cc233" (UID: "01456588-661b-424b-a6b0-3ebcca0cc233"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:38.234538 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.234492 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01456588-661b-424b-a6b0-3ebcca0cc233-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:54:38.234538 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.234530 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01456588-661b-424b-a6b0-3ebcca0cc233-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:54:38.234538 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.234542 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbbhc\" (UniqueName: \"kubernetes.io/projected/01456588-661b-424b-a6b0-3ebcca0cc233-kube-api-access-tbbhc\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:54:38.717118 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.715697 4468 generic.go:356] "Generic (PLEG): container finished" podID="01456588-661b-424b-a6b0-3ebcca0cc233" containerID="567ef5a6d6e53cd790a6ecbbf61236717b9f86f1020e2c51840ee733474e2305" exitCode=0 Oct 30 12:54:38.717118 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.715873 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" event={"ID":"01456588-661b-424b-a6b0-3ebcca0cc233","Type":"ContainerDied","Data":"567ef5a6d6e53cd790a6ecbbf61236717b9f86f1020e2c51840ee733474e2305"} Oct 30 12:54:38.717118 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.715971 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" event={"ID":"01456588-661b-424b-a6b0-3ebcca0cc233","Type":"ContainerDied","Data":"e6db66aa921631922feb90ceb619ee273aa2d626814589300cf7bf3c8e6653b1"} Oct 30 12:54:38.717118 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.716011 4468 scope.go:122] "RemoveContainer" containerID="567ef5a6d6e53cd790a6ecbbf61236717b9f86f1020e2c51840ee733474e2305" Oct 30 12:54:38.717118 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.716248 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv" Oct 30 12:54:38.734264 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.734239 4468 scope.go:122] "RemoveContainer" containerID="567ef5a6d6e53cd790a6ecbbf61236717b9f86f1020e2c51840ee733474e2305" Oct 30 12:54:38.734622 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:54:38.734591 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"567ef5a6d6e53cd790a6ecbbf61236717b9f86f1020e2c51840ee733474e2305\": container with ID starting with 567ef5a6d6e53cd790a6ecbbf61236717b9f86f1020e2c51840ee733474e2305 not found: ID does not exist" containerID="567ef5a6d6e53cd790a6ecbbf61236717b9f86f1020e2c51840ee733474e2305" Oct 30 12:54:38.734734 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.734629 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567ef5a6d6e53cd790a6ecbbf61236717b9f86f1020e2c51840ee733474e2305"} err="failed to get container status \"567ef5a6d6e53cd790a6ecbbf61236717b9f86f1020e2c51840ee733474e2305\": rpc error: code = NotFound desc = could not find container \"567ef5a6d6e53cd790a6ecbbf61236717b9f86f1020e2c51840ee733474e2305\": container with ID starting with 567ef5a6d6e53cd790a6ecbbf61236717b9f86f1020e2c51840ee733474e2305 not found: ID does not exist" Oct 30 12:54:38.779197 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.779161 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv"] Oct 30 12:54:38.784339 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:38.784303 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84m2lhv"] Oct 30 12:54:39.096056 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:39.095964 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01456588-661b-424b-a6b0-3ebcca0cc233" path="/var/lib/kubelet/pods/01456588-661b-424b-a6b0-3ebcca0cc233/volumes" Oct 30 12:54:41.019148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:41.019109 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn"] Oct 30 12:54:42.113951 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:42.113912 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf"] Oct 30 12:54:46.019578 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.019527 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" podUID="8f86a91b-53e8-4cee-95e8-086df48499e2" containerName="payload-processing" containerID="cri-o://287e32dc52ac7b454f1d69430246873b4cb6679baf3b3c325a45cce4f7660f5a" gracePeriod=25 Oct 30 12:54:46.307936 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.307905 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" Oct 30 12:54:46.433894 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.433858 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f86a91b-53e8-4cee-95e8-086df48499e2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f86a91b-53e8-4cee-95e8-086df48499e2-provider-credentials\") pod \"8f86a91b-53e8-4cee-95e8-086df48499e2\" (UID: \"8f86a91b-53e8-4cee-95e8-086df48499e2\") " Oct 30 12:54:46.434092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.434054 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f86a91b-53e8-4cee-95e8-086df48499e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f86a91b-53e8-4cee-95e8-086df48499e2-plugins-config-volume\") pod \"8f86a91b-53e8-4cee-95e8-086df48499e2\" (UID: \"8f86a91b-53e8-4cee-95e8-086df48499e2\") " Oct 30 12:54:46.434186 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.434172 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f86a91b-53e8-4cee-95e8-086df48499e2-kube-api-access-krqv2\" (UniqueName: \"kubernetes.io/projected/8f86a91b-53e8-4cee-95e8-086df48499e2-kube-api-access-krqv2\") pod \"8f86a91b-53e8-4cee-95e8-086df48499e2\" (UID: \"8f86a91b-53e8-4cee-95e8-086df48499e2\") " Oct 30 12:54:46.434231 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.434222 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f86a91b-53e8-4cee-95e8-086df48499e2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f86a91b-53e8-4cee-95e8-086df48499e2-routing-overlay-volume\") pod \"8f86a91b-53e8-4cee-95e8-086df48499e2\" (UID: \"8f86a91b-53e8-4cee-95e8-086df48499e2\") " Oct 30 12:54:46.434615 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.434570 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f86a91b-53e8-4cee-95e8-086df48499e2-plugins-config-volume" pod "8f86a91b-53e8-4cee-95e8-086df48499e2" (UID: "8f86a91b-53e8-4cee-95e8-086df48499e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:54:46.434615 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.434602 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f86a91b-53e8-4cee-95e8-086df48499e2-routing-overlay-volume" pod "8f86a91b-53e8-4cee-95e8-086df48499e2" (UID: "8f86a91b-53e8-4cee-95e8-086df48499e2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:54:46.434823 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.434810 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f86a91b-53e8-4cee-95e8-086df48499e2-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:54:46.434902 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.434828 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f86a91b-53e8-4cee-95e8-086df48499e2-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:54:46.436141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.436114 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f86a91b-53e8-4cee-95e8-086df48499e2-provider-credentials" pod "8f86a91b-53e8-4cee-95e8-086df48499e2" (UID: "8f86a91b-53e8-4cee-95e8-086df48499e2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:46.436322 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.436304 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f86a91b-53e8-4cee-95e8-086df48499e2-kube-api-access-krqv2" pod "8f86a91b-53e8-4cee-95e8-086df48499e2" (UID: "8f86a91b-53e8-4cee-95e8-086df48499e2"). InnerVolumeSpecName "kube-api-access-krqv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:46.536124 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.536028 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krqv2\" (UniqueName: \"kubernetes.io/projected/8f86a91b-53e8-4cee-95e8-086df48499e2-kube-api-access-krqv2\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:54:46.536124 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.536062 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f86a91b-53e8-4cee-95e8-086df48499e2-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:54:46.802380 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.802283 4468 generic.go:356] "Generic (PLEG): container finished" podID="8f86a91b-53e8-4cee-95e8-086df48499e2" containerID="287e32dc52ac7b454f1d69430246873b4cb6679baf3b3c325a45cce4f7660f5a" exitCode=0 Oct 30 12:54:46.802380 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.802332 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" Oct 30 12:54:46.802380 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.802364 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" event={"ID":"8f86a91b-53e8-4cee-95e8-086df48499e2","Type":"ContainerDied","Data":"287e32dc52ac7b454f1d69430246873b4cb6679baf3b3c325a45cce4f7660f5a"} Oct 30 12:54:46.802652 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.802424 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn" event={"ID":"8f86a91b-53e8-4cee-95e8-086df48499e2","Type":"ContainerDied","Data":"cbe1b08c869ffd9cb112b9e3ccb1ff37d13e1dee67f009f5877eeff5b95e29ee"} Oct 30 12:54:46.802652 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.802450 4468 scope.go:122] "RemoveContainer" containerID="287e32dc52ac7b454f1d69430246873b4cb6679baf3b3c325a45cce4f7660f5a" Oct 30 12:54:46.814016 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.813990 4468 scope.go:122] "RemoveContainer" containerID="287e32dc52ac7b454f1d69430246873b4cb6679baf3b3c325a45cce4f7660f5a" Oct 30 12:54:46.814356 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:54:46.814330 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287e32dc52ac7b454f1d69430246873b4cb6679baf3b3c325a45cce4f7660f5a\": container with ID starting with 287e32dc52ac7b454f1d69430246873b4cb6679baf3b3c325a45cce4f7660f5a not found: ID does not exist" containerID="287e32dc52ac7b454f1d69430246873b4cb6679baf3b3c325a45cce4f7660f5a" Oct 30 12:54:46.814467 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.814362 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287e32dc52ac7b454f1d69430246873b4cb6679baf3b3c325a45cce4f7660f5a"} err="failed to get container status \"287e32dc52ac7b454f1d69430246873b4cb6679baf3b3c325a45cce4f7660f5a\": rpc error: code = NotFound desc = could not find container \"287e32dc52ac7b454f1d69430246873b4cb6679baf3b3c325a45cce4f7660f5a\": container with ID starting with 287e32dc52ac7b454f1d69430246873b4cb6679baf3b3c325a45cce4f7660f5a not found: ID does not exist" Oct 30 12:54:46.873461 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.873419 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn"] Oct 30 12:54:46.887108 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:46.887057 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a786af/payload-processing-external-model-e2e-extmod-a-a786af-5677lv4vn"] Oct 30 12:54:47.116771 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.116668 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" podUID="9d47a9c7-345b-4111-bec3-a9a5801ba911" containerName="payload-processing" containerID="cri-o://a819be975bd1b3e102522fb0baa50319723ad7e8870415b495a66ea3c62700d0" gracePeriod=25 Oct 30 12:54:47.130962 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.130923 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f86a91b-53e8-4cee-95e8-086df48499e2" path="/var/lib/kubelet/pods/8f86a91b-53e8-4cee-95e8-086df48499e2/volumes" Oct 30 12:54:47.425497 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.425470 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" Oct 30 12:54:47.452433 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.452399 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d47a9c7-345b-4111-bec3-a9a5801ba911-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d47a9c7-345b-4111-bec3-a9a5801ba911-plugins-config-volume\") pod \"9d47a9c7-345b-4111-bec3-a9a5801ba911\" (UID: \"9d47a9c7-345b-4111-bec3-a9a5801ba911\") " Oct 30 12:54:47.452622 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.452459 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d47a9c7-345b-4111-bec3-a9a5801ba911-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d47a9c7-345b-4111-bec3-a9a5801ba911-provider-credentials\") pod \"9d47a9c7-345b-4111-bec3-a9a5801ba911\" (UID: \"9d47a9c7-345b-4111-bec3-a9a5801ba911\") " Oct 30 12:54:47.452622 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.452571 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d47a9c7-345b-4111-bec3-a9a5801ba911-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d47a9c7-345b-4111-bec3-a9a5801ba911-routing-overlay-volume\") pod \"9d47a9c7-345b-4111-bec3-a9a5801ba911\" (UID: \"9d47a9c7-345b-4111-bec3-a9a5801ba911\") " Oct 30 12:54:47.452622 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.452614 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d47a9c7-345b-4111-bec3-a9a5801ba911-kube-api-access-nbfzk\" (UniqueName: \"kubernetes.io/projected/9d47a9c7-345b-4111-bec3-a9a5801ba911-kube-api-access-nbfzk\") pod \"9d47a9c7-345b-4111-bec3-a9a5801ba911\" (UID: \"9d47a9c7-345b-4111-bec3-a9a5801ba911\") " Oct 30 12:54:47.452982 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.452944 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d47a9c7-345b-4111-bec3-a9a5801ba911-plugins-config-volume" pod "9d47a9c7-345b-4111-bec3-a9a5801ba911" (UID: "9d47a9c7-345b-4111-bec3-a9a5801ba911"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:54:47.453060 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.452976 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d47a9c7-345b-4111-bec3-a9a5801ba911-routing-overlay-volume" pod "9d47a9c7-345b-4111-bec3-a9a5801ba911" (UID: "9d47a9c7-345b-4111-bec3-a9a5801ba911"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:54:47.453444 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.453424 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d47a9c7-345b-4111-bec3-a9a5801ba911-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:54:47.453507 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.453450 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d47a9c7-345b-4111-bec3-a9a5801ba911-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:54:47.454893 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.454865 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d47a9c7-345b-4111-bec3-a9a5801ba911-provider-credentials" pod "9d47a9c7-345b-4111-bec3-a9a5801ba911" (UID: "9d47a9c7-345b-4111-bec3-a9a5801ba911"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:47.454994 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.454904 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d47a9c7-345b-4111-bec3-a9a5801ba911-kube-api-access-nbfzk" pod "9d47a9c7-345b-4111-bec3-a9a5801ba911" (UID: "9d47a9c7-345b-4111-bec3-a9a5801ba911"). InnerVolumeSpecName "kube-api-access-nbfzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:47.554585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.554541 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d47a9c7-345b-4111-bec3-a9a5801ba911-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:54:47.554585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.554578 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbfzk\" (UniqueName: \"kubernetes.io/projected/9d47a9c7-345b-4111-bec3-a9a5801ba911-kube-api-access-nbfzk\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:54:47.813912 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.813872 4468 generic.go:356] "Generic (PLEG): container finished" podID="9d47a9c7-345b-4111-bec3-a9a5801ba911" containerID="a819be975bd1b3e102522fb0baa50319723ad7e8870415b495a66ea3c62700d0" exitCode=0 Oct 30 12:54:47.814157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.813927 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" Oct 30 12:54:47.814157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.813959 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" event={"ID":"9d47a9c7-345b-4111-bec3-a9a5801ba911","Type":"ContainerDied","Data":"a819be975bd1b3e102522fb0baa50319723ad7e8870415b495a66ea3c62700d0"} Oct 30 12:54:47.814157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.814046 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf" event={"ID":"9d47a9c7-345b-4111-bec3-a9a5801ba911","Type":"ContainerDied","Data":"e465d3f37e79b4ad2aa700424d3a1ea79a0f17373736495d3c41526a4c4e65b1"} Oct 30 12:54:47.814157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.814085 4468 scope.go:122] "RemoveContainer" containerID="a819be975bd1b3e102522fb0baa50319723ad7e8870415b495a66ea3c62700d0" Oct 30 12:54:47.826693 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.826661 4468 scope.go:122] "RemoveContainer" containerID="a819be975bd1b3e102522fb0baa50319723ad7e8870415b495a66ea3c62700d0" Oct 30 12:54:47.827027 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:54:47.827007 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a819be975bd1b3e102522fb0baa50319723ad7e8870415b495a66ea3c62700d0\": container with ID starting with a819be975bd1b3e102522fb0baa50319723ad7e8870415b495a66ea3c62700d0 not found: ID does not exist" containerID="a819be975bd1b3e102522fb0baa50319723ad7e8870415b495a66ea3c62700d0" Oct 30 12:54:47.827136 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.827039 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a819be975bd1b3e102522fb0baa50319723ad7e8870415b495a66ea3c62700d0"} err="failed to get container status \"a819be975bd1b3e102522fb0baa50319723ad7e8870415b495a66ea3c62700d0\": rpc error: code = NotFound desc = could not find container \"a819be975bd1b3e102522fb0baa50319723ad7e8870415b495a66ea3c62700d0\": container with ID starting with a819be975bd1b3e102522fb0baa50319723ad7e8870415b495a66ea3c62700d0 not found: ID does not exist" Oct 30 12:54:47.874877 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.874836 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf"] Oct 30 12:54:47.886084 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.886036 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-6bf6rm5xf"] Oct 30 12:54:47.927536 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.927495 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c"] Oct 30 12:54:47.928288 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.928272 4468 cpu_manager.go:414] "containerMap: removing container" podUID="9d47a9c7-345b-4111-bec3-a9a5801ba911" containerName="payload-processing" Oct 30 12:54:47.928341 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.928288 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d47a9c7-345b-4111-bec3-a9a5801ba911" containerName="payload-processing" Oct 30 12:54:47.928341 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.928327 4468 cpu_manager.go:414] "containerMap: removing container" podUID="8f86a91b-53e8-4cee-95e8-086df48499e2" containerName="payload-processing" Oct 30 12:54:47.928341 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.928332 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f86a91b-53e8-4cee-95e8-086df48499e2" containerName="payload-processing" Oct 30 12:54:47.928442 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.928353 4468 cpu_manager.go:414] "containerMap: removing container" podUID="01456588-661b-424b-a6b0-3ebcca0cc233" containerName="payload-processing" Oct 30 12:54:47.928442 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.928358 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01456588-661b-424b-a6b0-3ebcca0cc233" containerName="payload-processing" Oct 30 12:54:47.928507 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.928460 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d47a9c7-345b-4111-bec3-a9a5801ba911" containerName="payload-processing" Oct 30 12:54:47.928507 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.928475 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f86a91b-53e8-4cee-95e8-086df48499e2" containerName="payload-processing" Oct 30 12:54:47.928507 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.928482 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="01456588-661b-424b-a6b0-3ebcca0cc233" containerName="payload-processing" Oct 30 12:54:47.933554 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.933531 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" Oct 30 12:54:47.954462 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.954420 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-63f4e4\"" Oct 30 12:54:47.954631 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.954521 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-63f4e4-dockercfg-bbmpm\"" Oct 30 12:54:47.961641 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:47.961603 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c"] Oct 30 12:54:48.150890 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:48.150682 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8"] Oct 30 12:54:48.156862 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:48.156828 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" Oct 30 12:54:48.159722 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:48.159581 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8"] Oct 30 12:54:48.184137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:48.176979 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add4979c-c091-4dea-9853-615756907859-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c\" (UID: \"add4979c-c091-4dea-9853-615756907859\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" Oct 30 12:54:48.184137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:48.177116 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416eb2ba-b5b4-4e25-a057-d870df27c1f7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8\" (UID: \"416eb2ba-b5b4-4e25-a057-d870df27c1f7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" Oct 30 12:54:48.184137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:48.177188 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4snpb\" (UniqueName: \"kubernetes.io/projected/add4979c-c091-4dea-9853-615756907859-kube-api-access-4snpb\") pod \"payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c\" (UID: \"add4979c-c091-4dea-9853-615756907859\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" Oct 30 12:54:48.184137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:48.177375 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqmqg\" (UniqueName: \"kubernetes.io/projected/416eb2ba-b5b4-4e25-a057-d870df27c1f7-kube-api-access-fqmqg\") pod \"payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8\" (UID: \"416eb2ba-b5b4-4e25-a057-d870df27c1f7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" Oct 30 12:54:48.279172 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:48.279133 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4snpb\" (UniqueName: \"kubernetes.io/projected/add4979c-c091-4dea-9853-615756907859-kube-api-access-4snpb\") pod \"payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c\" (UID: \"add4979c-c091-4dea-9853-615756907859\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" Oct 30 12:54:48.279361 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:48.279268 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqmqg\" (UniqueName: \"kubernetes.io/projected/416eb2ba-b5b4-4e25-a057-d870df27c1f7-kube-api-access-fqmqg\") pod \"payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8\" (UID: \"416eb2ba-b5b4-4e25-a057-d870df27c1f7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" Oct 30 12:54:48.279361 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:48.279350 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add4979c-c091-4dea-9853-615756907859-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c\" (UID: \"add4979c-c091-4dea-9853-615756907859\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" Oct 30 12:54:48.279453 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:48.279401 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416eb2ba-b5b4-4e25-a057-d870df27c1f7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8\" (UID: \"416eb2ba-b5b4-4e25-a057-d870df27c1f7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" Oct 30 12:54:48.280246 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:48.280220 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add4979c-c091-4dea-9853-615756907859-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c\" (UID: \"add4979c-c091-4dea-9853-615756907859\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" Oct 30 12:54:48.280345 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:48.280281 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416eb2ba-b5b4-4e25-a057-d870df27c1f7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8\" (UID: \"416eb2ba-b5b4-4e25-a057-d870df27c1f7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" Oct 30 12:54:48.306362 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:48.306331 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqmqg\" (UniqueName: \"kubernetes.io/projected/416eb2ba-b5b4-4e25-a057-d870df27c1f7-kube-api-access-fqmqg\") pod \"payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8\" (UID: \"416eb2ba-b5b4-4e25-a057-d870df27c1f7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" Oct 30 12:54:48.306624 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:48.306378 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4snpb\" (UniqueName: \"kubernetes.io/projected/add4979c-c091-4dea-9853-615756907859-kube-api-access-4snpb\") pod \"payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c\" (UID: \"add4979c-c091-4dea-9853-615756907859\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" Oct 30 12:54:48.494757 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:48.494708 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" Oct 30 12:54:48.549676 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:48.549638 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" Oct 30 12:54:48.819602 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:54:48.819563 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416eb2ba_b5b4_4e25_a057_d870df27c1f7.slice/crio-fc8dc2d0d7d4437cf976af28cd8e66b90d6658368a830316ec7afb199163905f WatchSource:0}: Error finding container fc8dc2d0d7d4437cf976af28cd8e66b90d6658368a830316ec7afb199163905f: Status 404 returned error can't find the container with id fc8dc2d0d7d4437cf976af28cd8e66b90d6658368a830316ec7afb199163905f Oct 30 12:54:48.823917 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:48.823893 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8"] Oct 30 12:54:48.882439 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:48.882407 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c"] Oct 30 12:54:48.888203 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:54:48.888167 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd4979c_c091_4dea_9853_615756907859.slice/crio-765c7692ea2a92fdb14899acfcbe9e25cfeb7d5ff0feb45e754f6a5a89bec90c WatchSource:0}: Error finding container 765c7692ea2a92fdb14899acfcbe9e25cfeb7d5ff0feb45e754f6a5a89bec90c: Status 404 returned error can't find the container with id 765c7692ea2a92fdb14899acfcbe9e25cfeb7d5ff0feb45e754f6a5a89bec90c Oct 30 12:54:49.101553 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:49.101469 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d47a9c7-345b-4111-bec3-a9a5801ba911" path="/var/lib/kubelet/pods/9d47a9c7-345b-4111-bec3-a9a5801ba911/volumes" Oct 30 12:54:49.832137 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:49.832096 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" event={"ID":"add4979c-c091-4dea-9853-615756907859","Type":"ContainerStarted","Data":"0acc223ed0522d6bf07bb554761774d2b4fc6b6affa85511869c8ee557f31800"} Oct 30 12:54:49.832650 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:49.832164 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" event={"ID":"add4979c-c091-4dea-9853-615756907859","Type":"ContainerStarted","Data":"765c7692ea2a92fdb14899acfcbe9e25cfeb7d5ff0feb45e754f6a5a89bec90c"} Oct 30 12:54:49.832650 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:49.832196 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" Oct 30 12:54:49.833912 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:49.833885 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" event={"ID":"416eb2ba-b5b4-4e25-a057-d870df27c1f7","Type":"ContainerStarted","Data":"8414499e476289550a8fc4700a95df4b656895b3805ba68932d10402cba1545d"} Oct 30 12:54:49.834010 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:49.833934 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" event={"ID":"416eb2ba-b5b4-4e25-a057-d870df27c1f7","Type":"ContainerStarted","Data":"fc8dc2d0d7d4437cf976af28cd8e66b90d6658368a830316ec7afb199163905f"} Oct 30 12:54:49.834010 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:49.833959 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" Oct 30 12:54:49.882532 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:49.882461 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" podStartSLOduration=2.882440201 podStartE2EDuration="2.882440201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:49.877869325 +0000 UTC m=+3640.912007723" watchObservedRunningTime="2026-10-30 12:54:49.882440201 +0000 UTC m=+3640.916578598" Oct 30 12:54:49.926455 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:49.926397 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" podStartSLOduration=2.926383279 podStartE2EDuration="2.926383279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:49.921560839 +0000 UTC m=+3640.955699235" watchObservedRunningTime="2026-10-30 12:54:49.926383279 +0000 UTC m=+3640.960521705" Oct 30 12:54:53.611917 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:54:53.611814 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:54:56.901748 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:56.901708 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs"] Oct 30 12:54:56.907869 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:56.907839 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" Oct 30 12:54:56.925242 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:56.925188 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv"] Oct 30 12:54:56.929918 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:56.929885 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6b67f8\"" Oct 30 12:54:56.930760 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:56.930738 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv" Oct 30 12:54:56.931082 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:56.931030 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs"] Oct 30 12:54:56.960124 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:56.952132 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv"] Oct 30 12:54:56.960124 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:56.952471 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6b67f8-dockercfg-87xtd\"" Oct 30 12:54:57.056100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.055361 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76mr5\" (UniqueName: \"kubernetes.io/projected/9f68b80b-1983-4fc9-907f-20653341e98a-kube-api-access-76mr5\") pod \"payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs\" (UID: \"9f68b80b-1983-4fc9-907f-20653341e98a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" Oct 30 12:54:57.056100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.055550 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f68b80b-1983-4fc9-907f-20653341e98a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs\" (UID: \"9f68b80b-1983-4fc9-907f-20653341e98a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" Oct 30 12:54:57.056100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.055634 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e29344f-41fb-4929-aee9-739cf692fa68-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv\" (UID: \"6e29344f-41fb-4929-aee9-739cf692fa68\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv" Oct 30 12:54:57.056100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.055705 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm4xn\" (UniqueName: \"kubernetes.io/projected/6e29344f-41fb-4929-aee9-739cf692fa68-kube-api-access-hm4xn\") pod \"payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv\" (UID: \"6e29344f-41fb-4929-aee9-739cf692fa68\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv" Oct 30 12:54:57.165814 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.162084 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76mr5\" (UniqueName: \"kubernetes.io/projected/9f68b80b-1983-4fc9-907f-20653341e98a-kube-api-access-76mr5\") pod \"payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs\" (UID: \"9f68b80b-1983-4fc9-907f-20653341e98a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" Oct 30 12:54:57.165814 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.162283 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f68b80b-1983-4fc9-907f-20653341e98a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs\" (UID: \"9f68b80b-1983-4fc9-907f-20653341e98a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" Oct 30 12:54:57.165814 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.162359 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e29344f-41fb-4929-aee9-739cf692fa68-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv\" (UID: \"6e29344f-41fb-4929-aee9-739cf692fa68\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv" Oct 30 12:54:57.165814 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.162422 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm4xn\" (UniqueName: \"kubernetes.io/projected/6e29344f-41fb-4929-aee9-739cf692fa68-kube-api-access-hm4xn\") pod \"payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv\" (UID: \"6e29344f-41fb-4929-aee9-739cf692fa68\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv" Oct 30 12:54:57.165814 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.163894 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f68b80b-1983-4fc9-907f-20653341e98a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs\" (UID: \"9f68b80b-1983-4fc9-907f-20653341e98a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" Oct 30 12:54:57.165814 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.164655 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e29344f-41fb-4929-aee9-739cf692fa68-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv\" (UID: \"6e29344f-41fb-4929-aee9-739cf692fa68\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv" Oct 30 12:54:57.190967 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.190935 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76mr5\" (UniqueName: \"kubernetes.io/projected/9f68b80b-1983-4fc9-907f-20653341e98a-kube-api-access-76mr5\") pod \"payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs\" (UID: \"9f68b80b-1983-4fc9-907f-20653341e98a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" Oct 30 12:54:57.199105 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.198809 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm4xn\" (UniqueName: \"kubernetes.io/projected/6e29344f-41fb-4929-aee9-739cf692fa68-kube-api-access-hm4xn\") pod \"payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv\" (UID: \"6e29344f-41fb-4929-aee9-739cf692fa68\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv" Oct 30 12:54:57.228569 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.228528 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" Oct 30 12:54:57.255361 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.255327 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv" Oct 30 12:54:57.548104 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:54:57.548055 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f68b80b_1983_4fc9_907f_20653341e98a.slice/crio-feb40f3b4d58d72e9f33a355b8ae7a80bf5754e4c1349fb81070f3d26edb35c1 WatchSource:0}: Error finding container feb40f3b4d58d72e9f33a355b8ae7a80bf5754e4c1349fb81070f3d26edb35c1: Status 404 returned error can't find the container with id feb40f3b4d58d72e9f33a355b8ae7a80bf5754e4c1349fb81070f3d26edb35c1 Oct 30 12:54:57.551894 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.551870 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs"] Oct 30 12:54:57.581661 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.581630 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv"] Oct 30 12:54:57.591757 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:54:57.591730 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e29344f_41fb_4929_aee9_739cf692fa68.slice/crio-77d4e6a59f3e60625690317684fb07204ec0add5dc840440ac26b08e85d3ca1b WatchSource:0}: Error finding container 77d4e6a59f3e60625690317684fb07204ec0add5dc840440ac26b08e85d3ca1b: Status 404 returned error can't find the container with id 77d4e6a59f3e60625690317684fb07204ec0add5dc840440ac26b08e85d3ca1b Oct 30 12:54:57.902532 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.902441 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv" event={"ID":"6e29344f-41fb-4929-aee9-739cf692fa68","Type":"ContainerStarted","Data":"615dcc27d96c7f6111bad8e313bca866ecde4b9b9ee31d78535b2c99de2238db"} Oct 30 12:54:57.902532 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.902494 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv" event={"ID":"6e29344f-41fb-4929-aee9-739cf692fa68","Type":"ContainerStarted","Data":"77d4e6a59f3e60625690317684fb07204ec0add5dc840440ac26b08e85d3ca1b"} Oct 30 12:54:57.903045 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.902530 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv" Oct 30 12:54:57.904528 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.904496 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" event={"ID":"9f68b80b-1983-4fc9-907f-20653341e98a","Type":"ContainerStarted","Data":"f3a38a1527d21d58ef88fd99cdee05e8f32f32c20e88846d5f6483a6edf25784"} Oct 30 12:54:57.904643 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.904555 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" event={"ID":"9f68b80b-1983-4fc9-907f-20653341e98a","Type":"ContainerStarted","Data":"feb40f3b4d58d72e9f33a355b8ae7a80bf5754e4c1349fb81070f3d26edb35c1"} Oct 30 12:54:57.904643 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.904584 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" Oct 30 12:54:57.954693 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.954627 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv" podStartSLOduration=1.954612752 podStartE2EDuration="1.954612752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:57.949330608 +0000 UTC m=+3648.983469004" watchObservedRunningTime="2026-10-30 12:54:57.954612752 +0000 UTC m=+3648.988751148" Oct 30 12:54:57.994454 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:54:57.994398 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" podStartSLOduration=1.9943835920000001 podStartE2EDuration="1.994383592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:57.992576478 +0000 UTC m=+3649.026714875" watchObservedRunningTime="2026-10-30 12:54:57.994383592 +0000 UTC m=+3649.028521988" Oct 30 12:55:00.864092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:00.855146 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" Oct 30 12:55:00.864092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:00.856501 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" Oct 30 12:55:08.917626 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:08.917593 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv" Oct 30 12:55:08.918158 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:08.917741 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" Oct 30 12:55:09.714485 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:09.714438 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn"] Oct 30 12:55:09.714766 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:09.714658 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" podUID="b3984ddb-e0d7-407d-93db-510c1bc61f0e" containerName="istio-proxy" containerID="cri-o://ba0fe44273da8c39d4c39927c0c375bf8f7c77c28befb8ae81823643fc7bebe4" gracePeriod=30 Oct 30 12:55:15.027763 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.027734 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:55:15.029097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.029043 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-workload-certs\") pod \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " Oct 30 12:55:15.029199 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.029100 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3984ddb-e0d7-407d-93db-510c1bc61f0e-kube-api-access-l4brs\" (UniqueName: \"kubernetes.io/projected/b3984ddb-e0d7-407d-93db-510c1bc61f0e-kube-api-access-l4brs\") pod \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " Oct 30 12:55:15.029199 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.029150 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-data\") pod \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " Oct 30 12:55:15.029297 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.029218 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-token\" (UniqueName: \"kubernetes.io/projected/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-token\") pod \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " Oct 30 12:55:15.029333 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.029312 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istiod-ca-cert\") pod \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " Oct 30 12:55:15.029333 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.029312 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-workload-certs" pod "b3984ddb-e0d7-407d-93db-510c1bc61f0e" (UID: "b3984ddb-e0d7-407d-93db-510c1bc61f0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:55:15.029416 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.029366 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-podinfo\") pod \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " Oct 30 12:55:15.029416 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.029387 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-credential-socket\") pod \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " Oct 30 12:55:15.029528 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.029418 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-envoy\") pod \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " Oct 30 12:55:15.029528 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.029473 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-data" pod "b3984ddb-e0d7-407d-93db-510c1bc61f0e" (UID: "b3984ddb-e0d7-407d-93db-510c1bc61f0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:55:15.029528 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.029514 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-workload-socket\") pod \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\" (UID: \"b3984ddb-e0d7-407d-93db-510c1bc61f0e\") " Oct 30 12:55:15.029717 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.029690 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-credential-socket" pod "b3984ddb-e0d7-407d-93db-510c1bc61f0e" (UID: "b3984ddb-e0d7-407d-93db-510c1bc61f0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:55:15.029785 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.029747 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istiod-ca-cert" pod "b3984ddb-e0d7-407d-93db-510c1bc61f0e" (UID: "b3984ddb-e0d7-407d-93db-510c1bc61f0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:55:15.029936 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.029909 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-workload-socket" pod "b3984ddb-e0d7-407d-93db-510c1bc61f0e" (UID: "b3984ddb-e0d7-407d-93db-510c1bc61f0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:55:15.030274 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.030251 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:55:15.030382 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.030276 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:55:15.030382 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.030290 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:55:15.030382 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.030302 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:55:15.030382 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.030314 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:55:15.031585 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.031553 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-token" pod "b3984ddb-e0d7-407d-93db-510c1bc61f0e" (UID: "b3984ddb-e0d7-407d-93db-510c1bc61f0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:55:15.031769 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.031753 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-podinfo" pod "b3984ddb-e0d7-407d-93db-510c1bc61f0e" (UID: "b3984ddb-e0d7-407d-93db-510c1bc61f0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:55:15.031904 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.031885 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-envoy" pod "b3984ddb-e0d7-407d-93db-510c1bc61f0e" (UID: "b3984ddb-e0d7-407d-93db-510c1bc61f0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:55:15.031973 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.031935 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3984ddb-e0d7-407d-93db-510c1bc61f0e-kube-api-access-l4brs" pod "b3984ddb-e0d7-407d-93db-510c1bc61f0e" (UID: "b3984ddb-e0d7-407d-93db-510c1bc61f0e"). InnerVolumeSpecName "kube-api-access-l4brs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:55:15.078143 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.078060 4468 generic.go:356] "Generic (PLEG): container finished" podID="b3984ddb-e0d7-407d-93db-510c1bc61f0e" containerID="ba0fe44273da8c39d4c39927c0c375bf8f7c77c28befb8ae81823643fc7bebe4" exitCode=0 Oct 30 12:55:15.078143 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.078125 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" Oct 30 12:55:15.078374 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.078100 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" event={"ID":"b3984ddb-e0d7-407d-93db-510c1bc61f0e","Type":"ContainerDied","Data":"ba0fe44273da8c39d4c39927c0c375bf8f7c77c28befb8ae81823643fc7bebe4"} Oct 30 12:55:15.078374 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.078250 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn" event={"ID":"b3984ddb-e0d7-407d-93db-510c1bc61f0e","Type":"ContainerDied","Data":"183bc8b05205ee81ea87bc46d4daa3d1f899dba64186428b4d2e75515eecc375"} Oct 30 12:55:15.078374 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.078283 4468 scope.go:122] "RemoveContainer" containerID="ba0fe44273da8c39d4c39927c0c375bf8f7c77c28befb8ae81823643fc7bebe4" Oct 30 12:55:15.090792 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.090766 4468 scope.go:122] "RemoveContainer" containerID="ba0fe44273da8c39d4c39927c0c375bf8f7c77c28befb8ae81823643fc7bebe4" Oct 30 12:55:15.091106 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:55:15.091063 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0fe44273da8c39d4c39927c0c375bf8f7c77c28befb8ae81823643fc7bebe4\": container with ID starting with ba0fe44273da8c39d4c39927c0c375bf8f7c77c28befb8ae81823643fc7bebe4 not found: ID does not exist" containerID="ba0fe44273da8c39d4c39927c0c375bf8f7c77c28befb8ae81823643fc7bebe4" Oct 30 12:55:15.091210 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.091116 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0fe44273da8c39d4c39927c0c375bf8f7c77c28befb8ae81823643fc7bebe4"} err="failed to get container status \"ba0fe44273da8c39d4c39927c0c375bf8f7c77c28befb8ae81823643fc7bebe4\": rpc error: code = NotFound desc = could not find container \"ba0fe44273da8c39d4c39927c0c375bf8f7c77c28befb8ae81823643fc7bebe4\": container with ID starting with ba0fe44273da8c39d4c39927c0c375bf8f7c77c28befb8ae81823643fc7bebe4 not found: ID does not exist" Oct 30 12:55:15.132772 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.132734 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:55:15.133221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.133171 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:55:15.133221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.133201 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4brs\" (UniqueName: \"kubernetes.io/projected/b3984ddb-e0d7-407d-93db-510c1bc61f0e-kube-api-access-l4brs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:55:15.133221 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.133215 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3984ddb-e0d7-407d-93db-510c1bc61f0e-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:55:15.173438 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.173397 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn"] Oct 30 12:55:15.183061 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:15.183022 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-6kdzn"] Oct 30 12:55:17.092269 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:17.092227 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3984ddb-e0d7-407d-93db-510c1bc61f0e" path="/var/lib/kubelet/pods/b3984ddb-e0d7-407d-93db-510c1bc61f0e/volumes" Oct 30 12:55:23.601157 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:55:23.601109 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:55:25.894053 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:25.894014 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv"] Oct 30 12:55:25.894908 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:25.894890 4468 cpu_manager.go:414] "containerMap: removing container" podUID="b3984ddb-e0d7-407d-93db-510c1bc61f0e" containerName="istio-proxy" Oct 30 12:55:25.894950 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:25.894910 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3984ddb-e0d7-407d-93db-510c1bc61f0e" containerName="istio-proxy" Oct 30 12:55:25.895227 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:25.895213 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3984ddb-e0d7-407d-93db-510c1bc61f0e" containerName="istio-proxy" Oct 30 12:55:25.900857 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:25.900835 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" Oct 30 12:55:25.923707 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:25.923654 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv"] Oct 30 12:55:25.930010 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:25.929959 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-57f1b32b361d\"" Oct 30 12:55:25.930240 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:25.930177 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-4ktzx\"" Oct 30 12:55:25.930390 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:25.930373 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 12:55:25.930513 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:25.930497 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 12:55:25.935594 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:25.935561 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 12:55:25.935594 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:25.935561 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 12:55:25.950114 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:25.950050 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2be98b1c-1f77-440f-8757-21e65bf5789a-routing-overlay-volume\") pod \"payload-processing-external-model-7f7cbc76ff-559nv\" (UID: \"2be98b1c-1f77-440f-8757-21e65bf5789a\") " pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" Oct 30 12:55:25.950439 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:25.950413 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be98b1c-1f77-440f-8757-21e65bf5789a-plugins-config-volume\") pod \"payload-processing-external-model-7f7cbc76ff-559nv\" (UID: \"2be98b1c-1f77-440f-8757-21e65bf5789a\") " pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" Oct 30 12:55:25.950571 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:25.950537 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2be98b1c-1f77-440f-8757-21e65bf5789a-provider-credentials\") pod \"payload-processing-external-model-7f7cbc76ff-559nv\" (UID: \"2be98b1c-1f77-440f-8757-21e65bf5789a\") " pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" Oct 30 12:55:25.950689 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:25.950671 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w22ql\" (UniqueName: \"kubernetes.io/projected/2be98b1c-1f77-440f-8757-21e65bf5789a-kube-api-access-w22ql\") pod \"payload-processing-external-model-7f7cbc76ff-559nv\" (UID: \"2be98b1c-1f77-440f-8757-21e65bf5789a\") " pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" Oct 30 12:55:26.052745 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:26.052695 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2be98b1c-1f77-440f-8757-21e65bf5789a-routing-overlay-volume\") pod \"payload-processing-external-model-7f7cbc76ff-559nv\" (UID: \"2be98b1c-1f77-440f-8757-21e65bf5789a\") " pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" Oct 30 12:55:26.052958 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:26.052782 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be98b1c-1f77-440f-8757-21e65bf5789a-plugins-config-volume\") pod \"payload-processing-external-model-7f7cbc76ff-559nv\" (UID: \"2be98b1c-1f77-440f-8757-21e65bf5789a\") " pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" Oct 30 12:55:26.052958 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:26.052825 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2be98b1c-1f77-440f-8757-21e65bf5789a-provider-credentials\") pod \"payload-processing-external-model-7f7cbc76ff-559nv\" (UID: \"2be98b1c-1f77-440f-8757-21e65bf5789a\") " pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" Oct 30 12:55:26.052958 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:26.052893 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w22ql\" (UniqueName: \"kubernetes.io/projected/2be98b1c-1f77-440f-8757-21e65bf5789a-kube-api-access-w22ql\") pod \"payload-processing-external-model-7f7cbc76ff-559nv\" (UID: \"2be98b1c-1f77-440f-8757-21e65bf5789a\") " pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" Oct 30 12:55:26.053577 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:26.053553 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2be98b1c-1f77-440f-8757-21e65bf5789a-routing-overlay-volume\") pod \"payload-processing-external-model-7f7cbc76ff-559nv\" (UID: \"2be98b1c-1f77-440f-8757-21e65bf5789a\") " pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" Oct 30 12:55:26.053726 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:26.053703 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be98b1c-1f77-440f-8757-21e65bf5789a-plugins-config-volume\") pod \"payload-processing-external-model-7f7cbc76ff-559nv\" (UID: \"2be98b1c-1f77-440f-8757-21e65bf5789a\") " pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" Oct 30 12:55:26.055423 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:26.055398 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2be98b1c-1f77-440f-8757-21e65bf5789a-provider-credentials\") pod \"payload-processing-external-model-7f7cbc76ff-559nv\" (UID: \"2be98b1c-1f77-440f-8757-21e65bf5789a\") " pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" Oct 30 12:55:26.080994 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:26.080958 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w22ql\" (UniqueName: \"kubernetes.io/projected/2be98b1c-1f77-440f-8757-21e65bf5789a-kube-api-access-w22ql\") pod \"payload-processing-external-model-7f7cbc76ff-559nv\" (UID: \"2be98b1c-1f77-440f-8757-21e65bf5789a\") " pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" Oct 30 12:55:26.218407 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:26.218352 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" Oct 30 12:55:26.540824 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:55:26.540783 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be98b1c_1f77_440f_8757_21e65bf5789a.slice/crio-b357ed219fcc9947d3a6bfd04b8402923947b9d29c847c8b364f41bf49ba49e9 WatchSource:0}: Error finding container b357ed219fcc9947d3a6bfd04b8402923947b9d29c847c8b364f41bf49ba49e9: Status 404 returned error can't find the container with id b357ed219fcc9947d3a6bfd04b8402923947b9d29c847c8b364f41bf49ba49e9 Oct 30 12:55:26.544602 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:26.544576 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv"] Oct 30 12:55:27.197317 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:27.197272 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" event={"ID":"2be98b1c-1f77-440f-8757-21e65bf5789a","Type":"ContainerStarted","Data":"471c35842d8d86bab1c64c9db91edff52ef38851c9cc685a6c05b3d79bbd4fb2"} Oct 30 12:55:27.197836 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:27.197344 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" event={"ID":"2be98b1c-1f77-440f-8757-21e65bf5789a","Type":"ContainerStarted","Data":"b357ed219fcc9947d3a6bfd04b8402923947b9d29c847c8b364f41bf49ba49e9"} Oct 30 12:55:27.198296 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:27.198279 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" Oct 30 12:55:27.245336 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:27.245284 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" podStartSLOduration=2.245270438 podStartE2EDuration="2.245270438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:27.244153845 +0000 UTC m=+3678.278292241" watchObservedRunningTime="2026-10-30 12:55:27.245270438 +0000 UTC m=+3678.279408834" Oct 30 12:55:33.300723 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:33.300678 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc"] Oct 30 12:55:33.306899 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:33.306869 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" Oct 30 12:55:33.336582 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:33.336530 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-63f4e4-serving-cert\"" Oct 30 12:55:33.339150 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:33.339079 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc"] Oct 30 12:55:33.436459 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:33.436420 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e977ddbb-e36d-4056-b775-d91dbbb91881-metrics-certs\") pod \"maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc\" (UID: \"e977ddbb-e36d-4056-b775-d91dbbb91881\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" Oct 30 12:55:33.436650 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:33.436606 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxhc6\" (UniqueName: \"kubernetes.io/projected/e977ddbb-e36d-4056-b775-d91dbbb91881-kube-api-access-bxhc6\") pod \"maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc\" (UID: \"e977ddbb-e36d-4056-b775-d91dbbb91881\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" Oct 30 12:55:33.436735 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:33.436649 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e977ddbb-e36d-4056-b775-d91dbbb91881-openshift-service-ca\") pod \"maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc\" (UID: \"e977ddbb-e36d-4056-b775-d91dbbb91881\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" Oct 30 12:55:33.436789 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:33.436746 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e977ddbb-e36d-4056-b775-d91dbbb91881-maas-api-tls\") pod \"maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc\" (UID: \"e977ddbb-e36d-4056-b775-d91dbbb91881\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" Oct 30 12:55:33.540795 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:33.539829 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxhc6\" (UniqueName: \"kubernetes.io/projected/e977ddbb-e36d-4056-b775-d91dbbb91881-kube-api-access-bxhc6\") pod \"maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc\" (UID: \"e977ddbb-e36d-4056-b775-d91dbbb91881\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" Oct 30 12:55:33.540795 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:33.539894 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e977ddbb-e36d-4056-b775-d91dbbb91881-openshift-service-ca\") pod \"maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc\" (UID: \"e977ddbb-e36d-4056-b775-d91dbbb91881\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" Oct 30 12:55:33.540795 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:33.539972 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e977ddbb-e36d-4056-b775-d91dbbb91881-maas-api-tls\") pod \"maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc\" (UID: \"e977ddbb-e36d-4056-b775-d91dbbb91881\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" Oct 30 12:55:33.540795 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:33.540154 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e977ddbb-e36d-4056-b775-d91dbbb91881-metrics-certs\") pod \"maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc\" (UID: \"e977ddbb-e36d-4056-b775-d91dbbb91881\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" Oct 30 12:55:33.543836 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:33.543807 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e977ddbb-e36d-4056-b775-d91dbbb91881-openshift-service-ca\") pod \"maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc\" (UID: \"e977ddbb-e36d-4056-b775-d91dbbb91881\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" Oct 30 12:55:33.565141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:33.559843 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e977ddbb-e36d-4056-b775-d91dbbb91881-metrics-certs\") pod \"maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc\" (UID: \"e977ddbb-e36d-4056-b775-d91dbbb91881\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" Oct 30 12:55:33.565141 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:33.563776 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e977ddbb-e36d-4056-b775-d91dbbb91881-maas-api-tls\") pod \"maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc\" (UID: \"e977ddbb-e36d-4056-b775-d91dbbb91881\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" Oct 30 12:55:33.570401 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:33.570358 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxhc6\" (UniqueName: \"kubernetes.io/projected/e977ddbb-e36d-4056-b775-d91dbbb91881-kube-api-access-bxhc6\") pod \"maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc\" (UID: \"e977ddbb-e36d-4056-b775-d91dbbb91881\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" Oct 30 12:55:33.629567 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:33.629523 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" Oct 30 12:55:33.944142 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:55:33.944105 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode977ddbb_e36d_4056_b775_d91dbbb91881.slice/crio-dffd58669961b36c3e50e50223790a98485eaeb98332424f1e0b965d52cc5802 WatchSource:0}: Error finding container dffd58669961b36c3e50e50223790a98485eaeb98332424f1e0b965d52cc5802: Status 404 returned error can't find the container with id dffd58669961b36c3e50e50223790a98485eaeb98332424f1e0b965d52cc5802 Oct 30 12:55:33.946113 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:33.946091 4468 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:55:33.948304 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:33.948277 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc"] Oct 30 12:55:34.259679 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.259589 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" event={"ID":"e977ddbb-e36d-4056-b775-d91dbbb91881","Type":"ContainerStarted","Data":"dffd58669961b36c3e50e50223790a98485eaeb98332424f1e0b965d52cc5802"} Oct 30 12:55:34.447968 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.447938 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs"] Oct 30 12:55:34.452979 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.452954 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" Oct 30 12:55:34.474967 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.474919 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs"] Oct 30 12:55:34.478735 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.478703 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1333c8\"" Oct 30 12:55:34.525385 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.525296 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5"] Oct 30 12:55:34.540087 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.539064 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5" Oct 30 12:55:34.544896 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.544851 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5"] Oct 30 12:55:34.556584 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.556547 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh5zq\" (UniqueName: \"kubernetes.io/projected/6d9cb487-201b-4bc1-ba55-c3e7c4306174-kube-api-access-lh5zq\") pod \"payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs\" (UID: \"6d9cb487-201b-4bc1-ba55-c3e7c4306174\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" Oct 30 12:55:34.556787 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.556615 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1333c8-dockercfg-8lg6m\"" Oct 30 12:55:34.556787 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.556714 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9cb487-201b-4bc1-ba55-c3e7c4306174-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs\" (UID: \"6d9cb487-201b-4bc1-ba55-c3e7c4306174\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" Oct 30 12:55:34.669093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.668045 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd5da4da-cc71-4a46-a697-396a96373776-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5\" (UID: \"cd5da4da-cc71-4a46-a697-396a96373776\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5" Oct 30 12:55:34.669093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.668190 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh5zq\" (UniqueName: \"kubernetes.io/projected/6d9cb487-201b-4bc1-ba55-c3e7c4306174-kube-api-access-lh5zq\") pod \"payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs\" (UID: \"6d9cb487-201b-4bc1-ba55-c3e7c4306174\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" Oct 30 12:55:34.669093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.668359 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9cb487-201b-4bc1-ba55-c3e7c4306174-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs\" (UID: \"6d9cb487-201b-4bc1-ba55-c3e7c4306174\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" Oct 30 12:55:34.669093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.668404 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtx4m\" (UniqueName: \"kubernetes.io/projected/cd5da4da-cc71-4a46-a697-396a96373776-kube-api-access-wtx4m\") pod \"payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5\" (UID: \"cd5da4da-cc71-4a46-a697-396a96373776\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5" Oct 30 12:55:34.700305 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.669773 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9cb487-201b-4bc1-ba55-c3e7c4306174-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs\" (UID: \"6d9cb487-201b-4bc1-ba55-c3e7c4306174\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" Oct 30 12:55:34.730189 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.729836 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh5zq\" (UniqueName: \"kubernetes.io/projected/6d9cb487-201b-4bc1-ba55-c3e7c4306174-kube-api-access-lh5zq\") pod \"payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs\" (UID: \"6d9cb487-201b-4bc1-ba55-c3e7c4306174\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" Oct 30 12:55:34.773104 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.770189 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtx4m\" (UniqueName: \"kubernetes.io/projected/cd5da4da-cc71-4a46-a697-396a96373776-kube-api-access-wtx4m\") pod \"payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5\" (UID: \"cd5da4da-cc71-4a46-a697-396a96373776\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5" Oct 30 12:55:34.773104 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.770380 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd5da4da-cc71-4a46-a697-396a96373776-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5\" (UID: \"cd5da4da-cc71-4a46-a697-396a96373776\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5" Oct 30 12:55:34.780148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.773789 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd5da4da-cc71-4a46-a697-396a96373776-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5\" (UID: \"cd5da4da-cc71-4a46-a697-396a96373776\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5" Oct 30 12:55:34.780148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.774137 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" Oct 30 12:55:34.813909 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.813875 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtx4m\" (UniqueName: \"kubernetes.io/projected/cd5da4da-cc71-4a46-a697-396a96373776-kube-api-access-wtx4m\") pod \"payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5\" (UID: \"cd5da4da-cc71-4a46-a697-396a96373776\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5" Oct 30 12:55:34.885176 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:34.884973 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5" Oct 30 12:55:35.121775 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:55:35.121719 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9cb487_201b_4bc1_ba55_c3e7c4306174.slice/crio-3b6760955af4c3ccc0d51ff8d3e9213aa7a6f3651017bcdcec86989f8e97e53c WatchSource:0}: Error finding container 3b6760955af4c3ccc0d51ff8d3e9213aa7a6f3651017bcdcec86989f8e97e53c: Status 404 returned error can't find the container with id 3b6760955af4c3ccc0d51ff8d3e9213aa7a6f3651017bcdcec86989f8e97e53c Oct 30 12:55:35.124381 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:35.124347 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs"] Oct 30 12:55:35.228782 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:55:35.228753 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd5da4da_cc71_4a46_a697_396a96373776.slice/crio-e0048206b749574c16cc813b6c13980c7529ce8b98ea114069d35a4b871da1fc WatchSource:0}: Error finding container e0048206b749574c16cc813b6c13980c7529ce8b98ea114069d35a4b871da1fc: Status 404 returned error can't find the container with id e0048206b749574c16cc813b6c13980c7529ce8b98ea114069d35a4b871da1fc Oct 30 12:55:35.231736 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:35.231707 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5"] Oct 30 12:55:35.268690 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:35.268655 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" event={"ID":"e977ddbb-e36d-4056-b775-d91dbbb91881","Type":"ContainerStarted","Data":"1f0b1fb0f410a7fb4951338779f77bb5aafb40dabfe545db22523eb71cf5919b"} Oct 30 12:55:35.268871 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:35.268719 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" Oct 30 12:55:35.270325 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:35.270294 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" event={"ID":"6d9cb487-201b-4bc1-ba55-c3e7c4306174","Type":"ContainerStarted","Data":"7aa04a4d8bc3dfb0991a5bd27a2612dec6397e28f04ca4ffe0cf42f168c0739d"} Oct 30 12:55:35.270460 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:35.270345 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" event={"ID":"6d9cb487-201b-4bc1-ba55-c3e7c4306174","Type":"ContainerStarted","Data":"3b6760955af4c3ccc0d51ff8d3e9213aa7a6f3651017bcdcec86989f8e97e53c"} Oct 30 12:55:35.270460 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:35.270406 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" Oct 30 12:55:35.271764 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:35.271735 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5" event={"ID":"cd5da4da-cc71-4a46-a697-396a96373776","Type":"ContainerStarted","Data":"e0048206b749574c16cc813b6c13980c7529ce8b98ea114069d35a4b871da1fc"} Oct 30 12:55:35.319109 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:35.319040 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" podStartSLOduration=1.896292957 podStartE2EDuration="2.319024436s" totalImagesPullingTime="422.731479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:35.31425935 +0000 UTC m=+3686.348397748" watchObservedRunningTime="2026-10-30 12:55:35.319024436 +0000 UTC m=+3686.353162832" Oct 30 12:55:35.358678 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:35.358623 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" podStartSLOduration=1.358612768 podStartE2EDuration="1.358612768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:35.357498127 +0000 UTC m=+3686.391636524" watchObservedRunningTime="2026-10-30 12:55:35.358612768 +0000 UTC m=+3686.392751203" Oct 30 12:55:36.281231 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:36.281186 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5" event={"ID":"cd5da4da-cc71-4a46-a697-396a96373776","Type":"ContainerStarted","Data":"bf19e83d66b3710474592acb381359524013eb6a71b92b8df05f2a7ab754fa6e"} Oct 30 12:55:36.281737 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:36.281371 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5" Oct 30 12:55:39.220244 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:39.220212 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" Oct 30 12:55:39.288380 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:39.288322 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5" podStartSLOduration=5.288306662 podStartE2EDuration="5.288306662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:36.339116826 +0000 UTC m=+3687.373255216" watchObservedRunningTime="2026-10-30 12:55:39.288306662 +0000 UTC m=+3690.322445058" Oct 30 12:55:41.286903 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:41.286870 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" Oct 30 12:55:46.285512 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:46.285475 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" Oct 30 12:55:47.293361 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:55:47.293328 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5" Oct 30 12:55:53.600572 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:55:53.600536 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:56:22.465678 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:22.465578 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r"] Oct 30 12:56:22.480249 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:22.480211 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" Oct 30 12:56:22.487220 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:22.487182 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r"] Oct 30 12:56:22.500308 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:22.498733 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1333c8-serving-cert\"" Oct 30 12:56:22.500308 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:22.498812 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7c9b69-2356-49ba-ad43-2a8773f48a46-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r\" (UID: \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" Oct 30 12:56:22.500308 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:22.498853 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7c9b69-2356-49ba-ad43-2a8773f48a46-maas-api-tls\") pod \"maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r\" (UID: \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" Oct 30 12:56:22.500308 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:22.498894 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bsvg\" (UniqueName: \"kubernetes.io/projected/7c7c9b69-2356-49ba-ad43-2a8773f48a46-kube-api-access-6bsvg\") pod \"maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r\" (UID: \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" Oct 30 12:56:22.500308 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:22.499043 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7c9b69-2356-49ba-ad43-2a8773f48a46-metrics-certs\") pod \"maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r\" (UID: \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" Oct 30 12:56:22.600502 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:22.600465 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7c9b69-2356-49ba-ad43-2a8773f48a46-metrics-certs\") pod \"maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r\" (UID: \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" Oct 30 12:56:22.600701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:22.600631 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7c9b69-2356-49ba-ad43-2a8773f48a46-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r\" (UID: \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" Oct 30 12:56:22.600701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:22.600657 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7c9b69-2356-49ba-ad43-2a8773f48a46-maas-api-tls\") pod \"maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r\" (UID: \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" Oct 30 12:56:22.600701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:22.600687 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bsvg\" (UniqueName: \"kubernetes.io/projected/7c7c9b69-2356-49ba-ad43-2a8773f48a46-kube-api-access-6bsvg\") pod \"maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r\" (UID: \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" Oct 30 12:56:22.601412 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:22.601389 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7c9b69-2356-49ba-ad43-2a8773f48a46-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r\" (UID: \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" Oct 30 12:56:22.603325 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:22.603302 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7c9b69-2356-49ba-ad43-2a8773f48a46-metrics-certs\") pod \"maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r\" (UID: \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" Oct 30 12:56:22.603432 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:22.603319 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7c9b69-2356-49ba-ad43-2a8773f48a46-maas-api-tls\") pod \"maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r\" (UID: \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" Oct 30 12:56:22.626912 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:22.626874 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bsvg\" (UniqueName: \"kubernetes.io/projected/7c7c9b69-2356-49ba-ad43-2a8773f48a46-kube-api-access-6bsvg\") pod \"maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r\" (UID: \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" Oct 30 12:56:22.817191 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:22.817102 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" Oct 30 12:56:23.126800 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:56:23.126766 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7c9b69_2356_49ba_ad43_2a8773f48a46.slice/crio-0d2389018f1b72476889be216160a2edadf1eb020b68371b75d2862826eaeadf WatchSource:0}: Error finding container 0d2389018f1b72476889be216160a2edadf1eb020b68371b75d2862826eaeadf: Status 404 returned error can't find the container with id 0d2389018f1b72476889be216160a2edadf1eb020b68371b75d2862826eaeadf Oct 30 12:56:23.129821 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:23.129797 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r"] Oct 30 12:56:23.606406 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:56:23.606371 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:56:23.784081 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:23.784036 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" event={"ID":"7c7c9b69-2356-49ba-ad43-2a8773f48a46","Type":"ContainerStarted","Data":"dbbb2d19c66178b0c82659b2db6f18de3fabeb760dd451646705ad61d54e27e2"} Oct 30 12:56:23.784234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:23.784122 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" event={"ID":"7c7c9b69-2356-49ba-ad43-2a8773f48a46","Type":"ContainerStarted","Data":"0d2389018f1b72476889be216160a2edadf1eb020b68371b75d2862826eaeadf"} Oct 30 12:56:23.784234 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:23.784157 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" Oct 30 12:56:23.831613 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:23.831513 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" podStartSLOduration=1.39246317 podStartE2EDuration="1.831498462s" totalImagesPullingTime="439.035292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:23.83029663 +0000 UTC m=+3734.864435025" watchObservedRunningTime="2026-10-30 12:56:23.831498462 +0000 UTC m=+3734.865636857" Oct 30 12:56:26.718928 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:26.718886 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv"] Oct 30 12:56:29.798579 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:29.798549 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" Oct 30 12:56:30.122262 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.122151 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj"] Oct 30 12:56:30.127685 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.127653 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" Oct 30 12:56:30.147378 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.147335 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"routing-overlay\"" Oct 30 12:56:30.147378 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.147366 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"openshift-service-ca.crt\"" Oct 30 12:56:30.147378 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.147369 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj"] Oct 30 12:56:30.153299 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.153257 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"kube-root-ca.crt\"" Oct 30 12:56:30.153534 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.153314 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-f7464f\"" Oct 30 12:56:30.153654 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.153375 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"agc-cred-d346468d7da4\"" Oct 30 12:56:30.153772 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.153465 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f7464f\"/\"payload-processing-external-model-e2e--a9926bc2-dockercfg-tx5dz\"" Oct 30 12:56:30.306946 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.306907 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vgbd\" (UniqueName: \"kubernetes.io/projected/9002883e-d5ab-4331-b4fa-f3ae000c9784-kube-api-access-5vgbd\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj\" (UID: \"9002883e-d5ab-4331-b4fa-f3ae000c9784\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" Oct 30 12:56:30.307173 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.307084 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9002883e-d5ab-4331-b4fa-f3ae000c9784-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj\" (UID: \"9002883e-d5ab-4331-b4fa-f3ae000c9784\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" Oct 30 12:56:30.307259 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.307228 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9002883e-d5ab-4331-b4fa-f3ae000c9784-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj\" (UID: \"9002883e-d5ab-4331-b4fa-f3ae000c9784\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" Oct 30 12:56:30.307327 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.307276 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9002883e-d5ab-4331-b4fa-f3ae000c9784-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj\" (UID: \"9002883e-d5ab-4331-b4fa-f3ae000c9784\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" Oct 30 12:56:30.414091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.410686 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vgbd\" (UniqueName: \"kubernetes.io/projected/9002883e-d5ab-4331-b4fa-f3ae000c9784-kube-api-access-5vgbd\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj\" (UID: \"9002883e-d5ab-4331-b4fa-f3ae000c9784\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" Oct 30 12:56:30.414091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.410847 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9002883e-d5ab-4331-b4fa-f3ae000c9784-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj\" (UID: \"9002883e-d5ab-4331-b4fa-f3ae000c9784\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" Oct 30 12:56:30.414091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.410921 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9002883e-d5ab-4331-b4fa-f3ae000c9784-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj\" (UID: \"9002883e-d5ab-4331-b4fa-f3ae000c9784\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" Oct 30 12:56:30.414091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.410953 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9002883e-d5ab-4331-b4fa-f3ae000c9784-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj\" (UID: \"9002883e-d5ab-4331-b4fa-f3ae000c9784\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" Oct 30 12:56:30.414091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.411843 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9002883e-d5ab-4331-b4fa-f3ae000c9784-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj\" (UID: \"9002883e-d5ab-4331-b4fa-f3ae000c9784\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" Oct 30 12:56:30.414091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.412452 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9002883e-d5ab-4331-b4fa-f3ae000c9784-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj\" (UID: \"9002883e-d5ab-4331-b4fa-f3ae000c9784\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" Oct 30 12:56:30.435093 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.433917 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9002883e-d5ab-4331-b4fa-f3ae000c9784-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj\" (UID: \"9002883e-d5ab-4331-b4fa-f3ae000c9784\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" Oct 30 12:56:30.446718 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.446682 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vgbd\" (UniqueName: \"kubernetes.io/projected/9002883e-d5ab-4331-b4fa-f3ae000c9784-kube-api-access-5vgbd\") pod \"payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj\" (UID: \"9002883e-d5ab-4331-b4fa-f3ae000c9784\") " pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" Oct 30 12:56:30.447622 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.447600 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" Oct 30 12:56:30.816482 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:56:30.816443 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9002883e_d5ab_4331_b4fa_f3ae000c9784.slice/crio-18e48fbc11a81a937f1e6f192007105d1576a9bb561f261291a3196caa6ee4e3 WatchSource:0}: Error finding container 18e48fbc11a81a937f1e6f192007105d1576a9bb561f261291a3196caa6ee4e3: Status 404 returned error can't find the container with id 18e48fbc11a81a937f1e6f192007105d1576a9bb561f261291a3196caa6ee4e3 Oct 30 12:56:30.819790 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.819766 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj"] Oct 30 12:56:30.857208 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:30.857164 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" event={"ID":"9002883e-d5ab-4331-b4fa-f3ae000c9784","Type":"ContainerStarted","Data":"18e48fbc11a81a937f1e6f192007105d1576a9bb561f261291a3196caa6ee4e3"} Oct 30 12:56:31.719673 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.719623 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" podUID="2be98b1c-1f77-440f-8757-21e65bf5789a" containerName="payload-processing" containerID="cri-o://471c35842d8d86bab1c64c9db91edff52ef38851c9cc685a6c05b3d79bbd4fb2" gracePeriod=25 Oct 30 12:56:31.797255 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.797213 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb"] Oct 30 12:56:31.802345 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.802319 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" Oct 30 12:56:31.844100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.834765 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ecb3aa\"/\"routing-overlay\"" Oct 30 12:56:31.844100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.834867 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ecb3aa\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ecb3aa\"" Oct 30 12:56:31.844100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.834930 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ecb3aa\"/\"kube-root-ca.crt\"" Oct 30 12:56:31.844100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.834999 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ecb3aa\"/\"agc-cred-904dc4da31c5\"" Oct 30 12:56:31.844100 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.835045 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ecb3aa\"/\"openshift-service-ca.crt\"" Oct 30 12:56:31.846166 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.846129 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9ddb0a-51f8-4d38-9605-74dc25689e46-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb\" (UID: \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" Oct 30 12:56:31.846313 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.846204 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7l5m\" (UniqueName: \"kubernetes.io/projected/0f9ddb0a-51f8-4d38-9605-74dc25689e46-kube-api-access-v7l5m\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb\" (UID: \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" Oct 30 12:56:31.846313 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.846301 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f9ddb0a-51f8-4d38-9605-74dc25689e46-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb\" (UID: \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" Oct 30 12:56:31.846449 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.846403 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f9ddb0a-51f8-4d38-9605-74dc25689e46-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb\" (UID: \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" Oct 30 12:56:31.863115 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.859312 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ecb3aa\"/\"payload-processing-external-model-e2e--8849cfcf-dockercfg-b62t2\"" Oct 30 12:56:31.936127 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.915634 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" event={"ID":"9002883e-d5ab-4331-b4fa-f3ae000c9784","Type":"ContainerStarted","Data":"a23ecf1e7f95ac6cec818d6d562e2411e64fabb26ac74f2100093953b1ecf265"} Oct 30 12:56:31.936127 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.916178 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" Oct 30 12:56:31.936127 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.922309 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb"] Oct 30 12:56:31.961119 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.955326 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f9ddb0a-51f8-4d38-9605-74dc25689e46-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb\" (UID: \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" Oct 30 12:56:31.961119 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.955520 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f9ddb0a-51f8-4d38-9605-74dc25689e46-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb\" (UID: \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" Oct 30 12:56:31.961119 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.955780 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9ddb0a-51f8-4d38-9605-74dc25689e46-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb\" (UID: \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" Oct 30 12:56:31.961119 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.955834 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7l5m\" (UniqueName: \"kubernetes.io/projected/0f9ddb0a-51f8-4d38-9605-74dc25689e46-kube-api-access-v7l5m\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb\" (UID: \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" Oct 30 12:56:31.961119 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.957141 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f9ddb0a-51f8-4d38-9605-74dc25689e46-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb\" (UID: \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" Oct 30 12:56:31.961119 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.959216 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9ddb0a-51f8-4d38-9605-74dc25689e46-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb\" (UID: \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" Oct 30 12:56:31.992087 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:31.990963 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" podStartSLOduration=1.9909432950000001 podStartE2EDuration="1.990943295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:31.989427523 +0000 UTC m=+3743.023565918" watchObservedRunningTime="2026-10-30 12:56:31.990943295 +0000 UTC m=+3743.025081695" Oct 30 12:56:32.005011 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.004302 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7l5m\" (UniqueName: \"kubernetes.io/projected/0f9ddb0a-51f8-4d38-9605-74dc25689e46-kube-api-access-v7l5m\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb\" (UID: \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" Oct 30 12:56:32.042387 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.041116 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f9ddb0a-51f8-4d38-9605-74dc25689e46-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb\" (UID: \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" Oct 30 12:56:32.204708 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.204663 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" Oct 30 12:56:32.246430 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.246347 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" Oct 30 12:56:32.364731 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.364665 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2be98b1c-1f77-440f-8757-21e65bf5789a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be98b1c-1f77-440f-8757-21e65bf5789a-plugins-config-volume\") pod \"2be98b1c-1f77-440f-8757-21e65bf5789a\" (UID: \"2be98b1c-1f77-440f-8757-21e65bf5789a\") " Oct 30 12:56:32.364731 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.364724 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2be98b1c-1f77-440f-8757-21e65bf5789a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2be98b1c-1f77-440f-8757-21e65bf5789a-provider-credentials\") pod \"2be98b1c-1f77-440f-8757-21e65bf5789a\" (UID: \"2be98b1c-1f77-440f-8757-21e65bf5789a\") " Oct 30 12:56:32.364983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.364813 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2be98b1c-1f77-440f-8757-21e65bf5789a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2be98b1c-1f77-440f-8757-21e65bf5789a-routing-overlay-volume\") pod \"2be98b1c-1f77-440f-8757-21e65bf5789a\" (UID: \"2be98b1c-1f77-440f-8757-21e65bf5789a\") " Oct 30 12:56:32.364983 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.364940 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2be98b1c-1f77-440f-8757-21e65bf5789a-kube-api-access-w22ql\" (UniqueName: \"kubernetes.io/projected/2be98b1c-1f77-440f-8757-21e65bf5789a-kube-api-access-w22ql\") pod \"2be98b1c-1f77-440f-8757-21e65bf5789a\" (UID: \"2be98b1c-1f77-440f-8757-21e65bf5789a\") " Oct 30 12:56:32.365541 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.365505 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2be98b1c-1f77-440f-8757-21e65bf5789a-plugins-config-volume" pod "2be98b1c-1f77-440f-8757-21e65bf5789a" (UID: "2be98b1c-1f77-440f-8757-21e65bf5789a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:56:32.366307 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.366277 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be98b1c-1f77-440f-8757-21e65bf5789a-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:56:32.366640 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.366610 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2be98b1c-1f77-440f-8757-21e65bf5789a-routing-overlay-volume" pod "2be98b1c-1f77-440f-8757-21e65bf5789a" (UID: "2be98b1c-1f77-440f-8757-21e65bf5789a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:56:32.368532 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.368504 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be98b1c-1f77-440f-8757-21e65bf5789a-provider-credentials" pod "2be98b1c-1f77-440f-8757-21e65bf5789a" (UID: "2be98b1c-1f77-440f-8757-21e65bf5789a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:56:32.368702 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.368616 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be98b1c-1f77-440f-8757-21e65bf5789a-kube-api-access-w22ql" pod "2be98b1c-1f77-440f-8757-21e65bf5789a" (UID: "2be98b1c-1f77-440f-8757-21e65bf5789a"). InnerVolumeSpecName "kube-api-access-w22ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:56:32.468233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.468196 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2be98b1c-1f77-440f-8757-21e65bf5789a-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:56:32.468233 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.468233 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w22ql\" (UniqueName: \"kubernetes.io/projected/2be98b1c-1f77-440f-8757-21e65bf5789a-kube-api-access-w22ql\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:56:32.468454 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.468248 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2be98b1c-1f77-440f-8757-21e65bf5789a-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 12:56:32.521597 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:56:32.521560 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9ddb0a_51f8_4d38_9605_74dc25689e46.slice/crio-5250b6a245885801663fd4470d6ee009c529d34890ee1287a4f3577755e7f816 WatchSource:0}: Error finding container 5250b6a245885801663fd4470d6ee009c529d34890ee1287a4f3577755e7f816: Status 404 returned error can't find the container with id 5250b6a245885801663fd4470d6ee009c529d34890ee1287a4f3577755e7f816 Oct 30 12:56:32.524588 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.524563 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb"] Oct 30 12:56:32.935889 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.935843 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" event={"ID":"0f9ddb0a-51f8-4d38-9605-74dc25689e46","Type":"ContainerStarted","Data":"1f2a869c9ebdb3080cf8aa02005d8feaae5eb879461a6a37c6e741c49026c1c9"} Oct 30 12:56:32.936420 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.935945 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" event={"ID":"0f9ddb0a-51f8-4d38-9605-74dc25689e46","Type":"ContainerStarted","Data":"5250b6a245885801663fd4470d6ee009c529d34890ee1287a4f3577755e7f816"} Oct 30 12:56:32.936420 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.936161 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" Oct 30 12:56:32.937507 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.937475 4468 generic.go:356] "Generic (PLEG): container finished" podID="2be98b1c-1f77-440f-8757-21e65bf5789a" containerID="471c35842d8d86bab1c64c9db91edff52ef38851c9cc685a6c05b3d79bbd4fb2" exitCode=0 Oct 30 12:56:32.937644 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.937617 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" event={"ID":"2be98b1c-1f77-440f-8757-21e65bf5789a","Type":"ContainerDied","Data":"471c35842d8d86bab1c64c9db91edff52ef38851c9cc685a6c05b3d79bbd4fb2"} Oct 30 12:56:32.937707 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.937626 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" Oct 30 12:56:32.937745 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.937708 4468 scope.go:122] "RemoveContainer" containerID="471c35842d8d86bab1c64c9db91edff52ef38851c9cc685a6c05b3d79bbd4fb2" Oct 30 12:56:32.937788 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.937696 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv" event={"ID":"2be98b1c-1f77-440f-8757-21e65bf5789a","Type":"ContainerDied","Data":"b357ed219fcc9947d3a6bfd04b8402923947b9d29c847c8b364f41bf49ba49e9"} Oct 30 12:56:32.949493 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.949471 4468 scope.go:122] "RemoveContainer" containerID="471c35842d8d86bab1c64c9db91edff52ef38851c9cc685a6c05b3d79bbd4fb2" Oct 30 12:56:32.949776 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:56:32.949753 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471c35842d8d86bab1c64c9db91edff52ef38851c9cc685a6c05b3d79bbd4fb2\": container with ID starting with 471c35842d8d86bab1c64c9db91edff52ef38851c9cc685a6c05b3d79bbd4fb2 not found: ID does not exist" containerID="471c35842d8d86bab1c64c9db91edff52ef38851c9cc685a6c05b3d79bbd4fb2" Oct 30 12:56:32.949877 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.949785 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471c35842d8d86bab1c64c9db91edff52ef38851c9cc685a6c05b3d79bbd4fb2"} err="failed to get container status \"471c35842d8d86bab1c64c9db91edff52ef38851c9cc685a6c05b3d79bbd4fb2\": rpc error: code = NotFound desc = could not find container \"471c35842d8d86bab1c64c9db91edff52ef38851c9cc685a6c05b3d79bbd4fb2\": container with ID starting with 471c35842d8d86bab1c64c9db91edff52ef38851c9cc685a6c05b3d79bbd4fb2 not found: ID does not exist" Oct 30 12:56:32.989561 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:32.989516 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" podStartSLOduration=1.989502759 podStartE2EDuration="1.989502759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:32.98425503 +0000 UTC m=+3744.018393425" watchObservedRunningTime="2026-10-30 12:56:32.989502759 +0000 UTC m=+3744.023641155" Oct 30 12:56:33.034198 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:33.034167 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv"] Oct 30 12:56:33.039279 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:33.039253 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f7cbc76ff-559nv"] Oct 30 12:56:33.093304 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:33.093231 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2be98b1c-1f77-440f-8757-21e65bf5789a" path="/var/lib/kubelet/pods/2be98b1c-1f77-440f-8757-21e65bf5789a/volumes" Oct 30 12:56:40.360257 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:40.360216 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r"] Oct 30 12:56:40.361674 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:40.361646 4468 cpu_manager.go:414] "containerMap: removing container" podUID="2be98b1c-1f77-440f-8757-21e65bf5789a" containerName="payload-processing" Oct 30 12:56:40.361674 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:40.361674 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2be98b1c-1f77-440f-8757-21e65bf5789a" containerName="payload-processing" Oct 30 12:56:40.361880 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:40.361859 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="2be98b1c-1f77-440f-8757-21e65bf5789a" containerName="payload-processing" Oct 30 12:56:40.375649 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:40.375619 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" Oct 30 12:56:40.401349 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:40.396772 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r"] Oct 30 12:56:40.462097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:40.457373 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-734ffe\"" Oct 30 12:56:40.462097 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:40.457538 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-734ffe-dockercfg-vmj47\"" Oct 30 12:56:40.540090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:40.528225 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xxbn\" (UniqueName: \"kubernetes.io/projected/49719ca9-b7db-4fd3-bcc8-feba390c6874-kube-api-access-6xxbn\") pod \"payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r\" (UID: \"49719ca9-b7db-4fd3-bcc8-feba390c6874\") " pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" Oct 30 12:56:40.540090 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:40.528356 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49719ca9-b7db-4fd3-bcc8-feba390c6874-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r\" (UID: \"49719ca9-b7db-4fd3-bcc8-feba390c6874\") " pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" Oct 30 12:56:40.630148 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:40.630009 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xxbn\" (UniqueName: \"kubernetes.io/projected/49719ca9-b7db-4fd3-bcc8-feba390c6874-kube-api-access-6xxbn\") pod \"payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r\" (UID: \"49719ca9-b7db-4fd3-bcc8-feba390c6874\") " pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" Oct 30 12:56:40.630331 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:40.630183 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49719ca9-b7db-4fd3-bcc8-feba390c6874-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r\" (UID: \"49719ca9-b7db-4fd3-bcc8-feba390c6874\") " pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" Oct 30 12:56:40.631168 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:40.631132 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49719ca9-b7db-4fd3-bcc8-feba390c6874-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r\" (UID: \"49719ca9-b7db-4fd3-bcc8-feba390c6874\") " pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" Oct 30 12:56:40.659663 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:40.659626 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xxbn\" (UniqueName: \"kubernetes.io/projected/49719ca9-b7db-4fd3-bcc8-feba390c6874-kube-api-access-6xxbn\") pod \"payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r\" (UID: \"49719ca9-b7db-4fd3-bcc8-feba390c6874\") " pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" Oct 30 12:56:40.702829 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:40.702790 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" Oct 30 12:56:41.050882 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:41.050754 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r"] Oct 30 12:56:42.022825 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:42.022788 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" event={"ID":"49719ca9-b7db-4fd3-bcc8-feba390c6874","Type":"ContainerStarted","Data":"03d0e084db11cfbf5acac856b70b115fb0c8f14b2335294caa22d1d54402acd6"} Oct 30 12:56:42.023377 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:42.022886 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" event={"ID":"49719ca9-b7db-4fd3-bcc8-feba390c6874","Type":"ContainerStarted","Data":"ef4d9ecf7b437c591917d626ebd1245c67dfac72d71e3289acd2a079fc4a42df"} Oct 30 12:56:42.023377 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:42.022917 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" Oct 30 12:56:42.068693 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:42.068616 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" podStartSLOduration=2.068595786 podStartE2EDuration="2.068595786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:42.06540503 +0000 UTC m=+3753.099543428" watchObservedRunningTime="2026-10-30 12:56:42.068595786 +0000 UTC m=+3753.102734182" Oct 30 12:56:42.941997 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:42.941962 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" Oct 30 12:56:43.949550 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:43.949520 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" Oct 30 12:56:49.278768 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:49.278727 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g"] Oct 30 12:56:49.283798 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:49.283776 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" Oct 30 12:56:49.304058 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:49.304029 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g"] Oct 30 12:56:49.309680 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:49.309649 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-befaa6\"" Oct 30 12:56:49.309862 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:49.309716 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-befaa6-dockercfg-tft48\"" Oct 30 12:56:49.366105 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:49.358564 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fe1ef7-c64e-4523-be4c-f5460fadf274-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g\" (UID: \"22fe1ef7-c64e-4523-be4c-f5460fadf274\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" Oct 30 12:56:49.366105 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:49.358783 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt6pl\" (UniqueName: \"kubernetes.io/projected/22fe1ef7-c64e-4523-be4c-f5460fadf274-kube-api-access-vt6pl\") pod \"payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g\" (UID: \"22fe1ef7-c64e-4523-be4c-f5460fadf274\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" Oct 30 12:56:49.515034 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:49.514996 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt6pl\" (UniqueName: \"kubernetes.io/projected/22fe1ef7-c64e-4523-be4c-f5460fadf274-kube-api-access-vt6pl\") pod \"payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g\" (UID: \"22fe1ef7-c64e-4523-be4c-f5460fadf274\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" Oct 30 12:56:49.515476 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:49.515451 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fe1ef7-c64e-4523-be4c-f5460fadf274-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g\" (UID: \"22fe1ef7-c64e-4523-be4c-f5460fadf274\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" Oct 30 12:56:49.516508 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:49.516484 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fe1ef7-c64e-4523-be4c-f5460fadf274-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g\" (UID: \"22fe1ef7-c64e-4523-be4c-f5460fadf274\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" Oct 30 12:56:49.543136 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:49.541616 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt6pl\" (UniqueName: \"kubernetes.io/projected/22fe1ef7-c64e-4523-be4c-f5460fadf274-kube-api-access-vt6pl\") pod \"payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g\" (UID: \"22fe1ef7-c64e-4523-be4c-f5460fadf274\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" Oct 30 12:56:49.605672 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:49.605637 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" Oct 30 12:56:49.961635 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:49.961179 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g"] Oct 30 12:56:50.195091 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:50.183532 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" event={"ID":"22fe1ef7-c64e-4523-be4c-f5460fadf274","Type":"ContainerStarted","Data":"aab019b204deee219bd861987658d9a5002b60212698fb498d9555459a963cb2"} Oct 30 12:56:51.195771 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:51.195728 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" event={"ID":"22fe1ef7-c64e-4523-be4c-f5460fadf274","Type":"ContainerStarted","Data":"52e87344ed7861777238f5ada60dcee04cf6355da16f5ccbe7e4293c5ef69dd6"} Oct 30 12:56:51.196276 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:51.196023 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" Oct 30 12:56:51.245017 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:51.244953 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" podStartSLOduration=2.24493479 podStartE2EDuration="2.24493479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:51.244708622 +0000 UTC m=+3762.278847020" watchObservedRunningTime="2026-10-30 12:56:51.24493479 +0000 UTC m=+3762.279073187" Oct 30 12:56:53.076095 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:56:53.075970 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" Oct 30 12:56:53.642442 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:56:53.642384 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:57:02.213893 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:02.213854 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" Oct 30 12:57:23.602412 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:57:23.602370 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:57:33.093496 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:33.093464 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk"] Oct 30 12:57:33.099905 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:33.099881 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" Oct 30 12:57:33.112342 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:33.112312 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk"] Oct 30 12:57:33.206639 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:33.205634 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9c3f12-dockercfg-49mwn\"" Oct 30 12:57:33.206639 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:33.205748 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9c3f12\"" Oct 30 12:57:33.219811 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:33.219780 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75rf6\" (UniqueName: \"kubernetes.io/projected/57f76bd7-cafb-48ad-9f99-8a7b88538a85-kube-api-access-75rf6\") pod \"payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk\" (UID: \"57f76bd7-cafb-48ad-9f99-8a7b88538a85\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" Oct 30 12:57:33.220054 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:33.220040 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f76bd7-cafb-48ad-9f99-8a7b88538a85-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk\" (UID: \"57f76bd7-cafb-48ad-9f99-8a7b88538a85\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" Oct 30 12:57:33.322157 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:33.322103 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f76bd7-cafb-48ad-9f99-8a7b88538a85-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk\" (UID: \"57f76bd7-cafb-48ad-9f99-8a7b88538a85\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" Oct 30 12:57:33.322594 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:33.322564 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75rf6\" (UniqueName: \"kubernetes.io/projected/57f76bd7-cafb-48ad-9f99-8a7b88538a85-kube-api-access-75rf6\") pod \"payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk\" (UID: \"57f76bd7-cafb-48ad-9f99-8a7b88538a85\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" Oct 30 12:57:33.323188 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:33.323020 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f76bd7-cafb-48ad-9f99-8a7b88538a85-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk\" (UID: \"57f76bd7-cafb-48ad-9f99-8a7b88538a85\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" Oct 30 12:57:33.348592 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:33.348518 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75rf6\" (UniqueName: \"kubernetes.io/projected/57f76bd7-cafb-48ad-9f99-8a7b88538a85-kube-api-access-75rf6\") pod \"payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk\" (UID: \"57f76bd7-cafb-48ad-9f99-8a7b88538a85\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" Oct 30 12:57:33.428490 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:33.428451 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" Oct 30 12:57:33.751260 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:57:33.751218 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f76bd7_cafb_48ad_9f99_8a7b88538a85.slice/crio-2be808856d113fbd6a0c5d7ed6daf04e7a6a1d6e6a13bfc6780e393af4f79b9b WatchSource:0}: Error finding container 2be808856d113fbd6a0c5d7ed6daf04e7a6a1d6e6a13bfc6780e393af4f79b9b: Status 404 returned error can't find the container with id 2be808856d113fbd6a0c5d7ed6daf04e7a6a1d6e6a13bfc6780e393af4f79b9b Oct 30 12:57:33.753516 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:33.753490 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk"] Oct 30 12:57:33.960502 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:33.960474 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4"] Oct 30 12:57:33.965385 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:33.965364 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" Oct 30 12:57:33.988806 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:33.986047 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-734ffe-serving-cert\"" Oct 30 12:57:33.992701 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:33.992287 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4"] Oct 30 12:57:34.149427 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:34.148973 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmhg9\" (UniqueName: \"kubernetes.io/projected/54a87f6f-513a-4a7b-a904-be6b4f218570-kube-api-access-cmhg9\") pod \"maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4\" (UID: \"54a87f6f-513a-4a7b-a904-be6b4f218570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" Oct 30 12:57:34.149427 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:34.149235 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54a87f6f-513a-4a7b-a904-be6b4f218570-metrics-certs\") pod \"maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4\" (UID: \"54a87f6f-513a-4a7b-a904-be6b4f218570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" Oct 30 12:57:34.149427 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:34.149347 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54a87f6f-513a-4a7b-a904-be6b4f218570-maas-api-tls\") pod \"maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4\" (UID: \"54a87f6f-513a-4a7b-a904-be6b4f218570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" Oct 30 12:57:34.150143 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:34.149469 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54a87f6f-513a-4a7b-a904-be6b4f218570-openshift-service-ca\") pod \"maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4\" (UID: \"54a87f6f-513a-4a7b-a904-be6b4f218570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" Oct 30 12:57:34.251294 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:34.251257 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54a87f6f-513a-4a7b-a904-be6b4f218570-metrics-certs\") pod \"maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4\" (UID: \"54a87f6f-513a-4a7b-a904-be6b4f218570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" Oct 30 12:57:34.251540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:34.251315 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54a87f6f-513a-4a7b-a904-be6b4f218570-maas-api-tls\") pod \"maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4\" (UID: \"54a87f6f-513a-4a7b-a904-be6b4f218570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" Oct 30 12:57:34.251540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:34.251407 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54a87f6f-513a-4a7b-a904-be6b4f218570-openshift-service-ca\") pod \"maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4\" (UID: \"54a87f6f-513a-4a7b-a904-be6b4f218570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" Oct 30 12:57:34.251540 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:34.251534 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmhg9\" (UniqueName: \"kubernetes.io/projected/54a87f6f-513a-4a7b-a904-be6b4f218570-kube-api-access-cmhg9\") pod \"maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4\" (UID: \"54a87f6f-513a-4a7b-a904-be6b4f218570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" Oct 30 12:57:34.252217 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:34.252191 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54a87f6f-513a-4a7b-a904-be6b4f218570-openshift-service-ca\") pod \"maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4\" (UID: \"54a87f6f-513a-4a7b-a904-be6b4f218570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" Oct 30 12:57:34.253867 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:34.253849 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54a87f6f-513a-4a7b-a904-be6b4f218570-maas-api-tls\") pod \"maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4\" (UID: \"54a87f6f-513a-4a7b-a904-be6b4f218570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" Oct 30 12:57:34.253931 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:34.253897 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54a87f6f-513a-4a7b-a904-be6b4f218570-metrics-certs\") pod \"maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4\" (UID: \"54a87f6f-513a-4a7b-a904-be6b4f218570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" Oct 30 12:57:34.278651 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:34.278624 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmhg9\" (UniqueName: \"kubernetes.io/projected/54a87f6f-513a-4a7b-a904-be6b4f218570-kube-api-access-cmhg9\") pod \"maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4\" (UID: \"54a87f6f-513a-4a7b-a904-be6b4f218570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" Oct 30 12:57:34.300466 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:34.300431 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" Oct 30 12:57:34.615138 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:57:34.615109 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a87f6f_513a_4a7b_a904_be6b4f218570.slice/crio-553b3497fc4e8507d4dccbfa14b3e76f5c8d24d10e9f80cebc4eb0c0df25c024 WatchSource:0}: Error finding container 553b3497fc4e8507d4dccbfa14b3e76f5c8d24d10e9f80cebc4eb0c0df25c024: Status 404 returned error can't find the container with id 553b3497fc4e8507d4dccbfa14b3e76f5c8d24d10e9f80cebc4eb0c0df25c024 Oct 30 12:57:34.619357 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:34.619332 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4"] Oct 30 12:57:34.679936 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:34.679902 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" event={"ID":"57f76bd7-cafb-48ad-9f99-8a7b88538a85","Type":"ContainerStarted","Data":"79c6e10614bdc667a132baa21bf3aed8416da9c91ed45f1ed6e01180d27d7972"} Oct 30 12:57:34.680124 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:34.679967 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" event={"ID":"57f76bd7-cafb-48ad-9f99-8a7b88538a85","Type":"ContainerStarted","Data":"2be808856d113fbd6a0c5d7ed6daf04e7a6a1d6e6a13bfc6780e393af4f79b9b"} Oct 30 12:57:34.680124 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:34.680000 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" Oct 30 12:57:34.681683 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:34.681662 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" event={"ID":"54a87f6f-513a-4a7b-a904-be6b4f218570","Type":"ContainerStarted","Data":"553b3497fc4e8507d4dccbfa14b3e76f5c8d24d10e9f80cebc4eb0c0df25c024"} Oct 30 12:57:34.729788 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:34.729733 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" podStartSLOduration=1.729719671 podStartE2EDuration="1.729719671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:34.728806763 +0000 UTC m=+3805.762945163" watchObservedRunningTime="2026-10-30 12:57:34.729719671 +0000 UTC m=+3805.763858067" Oct 30 12:57:35.692427 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:35.692385 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" event={"ID":"54a87f6f-513a-4a7b-a904-be6b4f218570","Type":"ContainerStarted","Data":"bfdb0b18005756cf21b5c931389fa31b272fd8f705027b5af65ab5117c9187a1"} Oct 30 12:57:35.692898 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:35.692566 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" Oct 30 12:57:35.740518 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:35.740459 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" podStartSLOduration=2.316444052 podStartE2EDuration="2.740447575s" totalImagesPullingTime="424.003523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:35.737542972 +0000 UTC m=+3806.771681369" watchObservedRunningTime="2026-10-30 12:57:35.740447575 +0000 UTC m=+3806.774585970" Oct 30 12:57:41.705863 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:41.705835 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" Oct 30 12:57:45.695745 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:57:45.695714 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" Oct 30 12:57:53.601634 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:57:53.601559 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:58:23.608192 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:58:23.608043 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:58:26.690923 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:26.690871 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb"] Oct 30 12:58:26.697553 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:26.697505 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" Oct 30 12:58:26.717153 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:26.717127 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9c3f12-serving-cert\"" Oct 30 12:58:26.726239 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:26.726212 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb"] Oct 30 12:58:26.817159 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:26.817112 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf35b28-b756-4281-80d7-bc361b2d56e3-metrics-certs\") pod \"maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb\" (UID: \"bcf35b28-b756-4281-80d7-bc361b2d56e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" Oct 30 12:58:26.817339 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:26.817175 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcf35b28-b756-4281-80d7-bc361b2d56e3-maas-api-tls\") pod \"maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb\" (UID: \"bcf35b28-b756-4281-80d7-bc361b2d56e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" Oct 30 12:58:26.817339 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:26.817300 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf35b28-b756-4281-80d7-bc361b2d56e3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb\" (UID: \"bcf35b28-b756-4281-80d7-bc361b2d56e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" Oct 30 12:58:26.817602 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:26.817583 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxhfk\" (UniqueName: \"kubernetes.io/projected/bcf35b28-b756-4281-80d7-bc361b2d56e3-kube-api-access-dxhfk\") pod \"maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb\" (UID: \"bcf35b28-b756-4281-80d7-bc361b2d56e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" Oct 30 12:58:26.922094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:26.920955 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxhfk\" (UniqueName: \"kubernetes.io/projected/bcf35b28-b756-4281-80d7-bc361b2d56e3-kube-api-access-dxhfk\") pod \"maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb\" (UID: \"bcf35b28-b756-4281-80d7-bc361b2d56e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" Oct 30 12:58:26.922094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:26.921239 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf35b28-b756-4281-80d7-bc361b2d56e3-metrics-certs\") pod \"maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb\" (UID: \"bcf35b28-b756-4281-80d7-bc361b2d56e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" Oct 30 12:58:26.922094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:26.921281 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcf35b28-b756-4281-80d7-bc361b2d56e3-maas-api-tls\") pod \"maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb\" (UID: \"bcf35b28-b756-4281-80d7-bc361b2d56e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" Oct 30 12:58:26.922094 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:26.921318 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf35b28-b756-4281-80d7-bc361b2d56e3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb\" (UID: \"bcf35b28-b756-4281-80d7-bc361b2d56e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" Oct 30 12:58:26.922454 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:26.922331 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf35b28-b756-4281-80d7-bc361b2d56e3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb\" (UID: \"bcf35b28-b756-4281-80d7-bc361b2d56e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" Oct 30 12:58:26.933092 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:26.925831 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf35b28-b756-4281-80d7-bc361b2d56e3-metrics-certs\") pod \"maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb\" (UID: \"bcf35b28-b756-4281-80d7-bc361b2d56e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" Oct 30 12:58:26.958778 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:26.957083 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcf35b28-b756-4281-80d7-bc361b2d56e3-maas-api-tls\") pod \"maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb\" (UID: \"bcf35b28-b756-4281-80d7-bc361b2d56e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" Oct 30 12:58:26.989947 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:26.989873 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxhfk\" (UniqueName: \"kubernetes.io/projected/bcf35b28-b756-4281-80d7-bc361b2d56e3-kube-api-access-dxhfk\") pod \"maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb\" (UID: \"bcf35b28-b756-4281-80d7-bc361b2d56e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" Oct 30 12:58:27.033578 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:27.033320 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" Oct 30 12:58:27.625706 ip-10-0-131-76 kubenswrapper[4468]: W1030 12:58:27.625673 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf35b28_b756_4281_80d7_bc361b2d56e3.slice/crio-830fd204a0bce340a80995f3872b82dc7b2f28a0c7af02a7eefb373596ac513b WatchSource:0}: Error finding container 830fd204a0bce340a80995f3872b82dc7b2f28a0c7af02a7eefb373596ac513b: Status 404 returned error can't find the container with id 830fd204a0bce340a80995f3872b82dc7b2f28a0c7af02a7eefb373596ac513b Oct 30 12:58:27.632145 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:27.631321 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb"] Oct 30 12:58:28.337199 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:28.336186 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" event={"ID":"bcf35b28-b756-4281-80d7-bc361b2d56e3","Type":"ContainerStarted","Data":"0eae3f80cdcd2815da50353fd8b07348ac0731d97aabb33f4696860c2dc4aeea"} Oct 30 12:58:28.337199 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:28.336271 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" event={"ID":"bcf35b28-b756-4281-80d7-bc361b2d56e3","Type":"ContainerStarted","Data":"830fd204a0bce340a80995f3872b82dc7b2f28a0c7af02a7eefb373596ac513b"} Oct 30 12:58:28.339764 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:28.338182 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" Oct 30 12:58:28.396865 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:28.396798 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" podStartSLOduration=2.010007744 podStartE2EDuration="2.396779615s" totalImagesPullingTime="386.771871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:28.385403527 +0000 UTC m=+3859.419541922" watchObservedRunningTime="2026-10-30 12:58:28.396779615 +0000 UTC m=+3859.430918011" Oct 30 12:58:35.359856 ip-10-0-131-76 kubenswrapper[4468]: I1030 12:58:35.359827 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" Oct 30 12:58:53.601595 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:58:53.601553 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:59:23.600817 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:59:23.600728 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 12:59:53.600541 ip-10-0-131-76 kubenswrapper[4468]: E1030 12:59:53.600499 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:00:00.260198 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.260163 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4"] Oct 30 13:00:00.265566 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.265545 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" Oct 30 13:00:00.279534 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.279499 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq"] Oct 30 13:00:00.289478 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.289453 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" Oct 30 13:00:00.293296 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.293274 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4"] Oct 30 13:00:00.298341 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.298319 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:00:00.313460 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.313437 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8"] Oct 30 13:00:00.323461 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.323436 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq"] Oct 30 13:00:00.323609 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.323551 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" Oct 30 13:00:00.331971 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.331949 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw"] Oct 30 13:00:00.337672 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.337648 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" Oct 30 13:00:00.346865 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.346802 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8"] Oct 30 13:00:00.351825 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.351799 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7rqx\" (UniqueName: \"kubernetes.io/projected/3314d4c4-7905-42dd-b342-277da538ba35-kube-api-access-s7rqx\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq\" (UID: \"3314d4c4-7905-42dd-b342-277da538ba35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" Oct 30 13:00:00.357413 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.357389 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4\" (UID: \"6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" Oct 30 13:00:00.357607 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.357587 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z7h8\" (UniqueName: \"kubernetes.io/projected/6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778-kube-api-access-9z7h8\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4\" (UID: \"6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" Oct 30 13:00:00.357814 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.357790 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3314d4c4-7905-42dd-b342-277da538ba35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq\" (UID: \"3314d4c4-7905-42dd-b342-277da538ba35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" Oct 30 13:00:00.369112 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.367397 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb"] Oct 30 13:00:00.373608 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.373586 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw"] Oct 30 13:00:00.373843 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.373829 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" Oct 30 13:00:00.379061 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.379041 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st"] Oct 30 13:00:00.388739 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.385288 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" Oct 30 13:00:00.388739 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.387636 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb"] Oct 30 13:00:00.395626 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.395597 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st"] Oct 30 13:00:00.459033 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.459000 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcjjz\" (UniqueName: \"kubernetes.io/projected/137214d6-09d8-4a4f-bf37-fcef28d37d07-kube-api-access-hcjjz\") pod \"maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw\" (UID: \"137214d6-09d8-4a4f-bf37-fcef28d37d07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" Oct 30 13:00:00.459254 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.459136 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de938051-680d-48df-80b7-8e71ee81217b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8\" (UID: \"de938051-680d-48df-80b7-8e71ee81217b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" Oct 30 13:00:00.459254 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.459236 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee548a01-d0b9-4387-a699-041ac6e9b205-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st\" (UID: \"ee548a01-d0b9-4387-a699-041ac6e9b205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" Oct 30 13:00:00.459370 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.459267 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4\" (UID: \"6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" Oct 30 13:00:00.459370 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.459302 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z7h8\" (UniqueName: \"kubernetes.io/projected/6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778-kube-api-access-9z7h8\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4\" (UID: \"6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" Oct 30 13:00:00.459470 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.459376 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvl9d\" (UniqueName: \"kubernetes.io/projected/9eb0a51c-3364-487b-8aec-77c2cb215dd7-kube-api-access-qvl9d\") pod \"maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb\" (UID: \"9eb0a51c-3364-487b-8aec-77c2cb215dd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" Oct 30 13:00:00.459470 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.459460 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3314d4c4-7905-42dd-b342-277da538ba35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq\" (UID: \"3314d4c4-7905-42dd-b342-277da538ba35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" Oct 30 13:00:00.459581 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.459512 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/137214d6-09d8-4a4f-bf37-fcef28d37d07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw\" (UID: \"137214d6-09d8-4a4f-bf37-fcef28d37d07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" Oct 30 13:00:00.459667 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.459635 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7rqx\" (UniqueName: \"kubernetes.io/projected/3314d4c4-7905-42dd-b342-277da538ba35-kube-api-access-s7rqx\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq\" (UID: \"3314d4c4-7905-42dd-b342-277da538ba35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" Oct 30 13:00:00.459743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.459671 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54j5l\" (UniqueName: \"kubernetes.io/projected/ee548a01-d0b9-4387-a699-041ac6e9b205-kube-api-access-54j5l\") pod \"maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st\" (UID: \"ee548a01-d0b9-4387-a699-041ac6e9b205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" Oct 30 13:00:00.459743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.459722 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eb0a51c-3364-487b-8aec-77c2cb215dd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb\" (UID: \"9eb0a51c-3364-487b-8aec-77c2cb215dd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" Oct 30 13:00:00.459860 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.459803 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qncwd\" (UniqueName: \"kubernetes.io/projected/de938051-680d-48df-80b7-8e71ee81217b-kube-api-access-qncwd\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8\" (UID: \"de938051-680d-48df-80b7-8e71ee81217b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" Oct 30 13:00:00.459972 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.459950 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4\" (UID: \"6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" Oct 30 13:00:00.460350 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.460327 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3314d4c4-7905-42dd-b342-277da538ba35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq\" (UID: \"3314d4c4-7905-42dd-b342-277da538ba35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" Oct 30 13:00:00.484363 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.484341 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z7h8\" (UniqueName: \"kubernetes.io/projected/6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778-kube-api-access-9z7h8\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4\" (UID: \"6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" Oct 30 13:00:00.484601 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.484581 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7rqx\" (UniqueName: \"kubernetes.io/projected/3314d4c4-7905-42dd-b342-277da538ba35-kube-api-access-s7rqx\") pod \"maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq\" (UID: \"3314d4c4-7905-42dd-b342-277da538ba35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" Oct 30 13:00:00.560911 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.560831 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54j5l\" (UniqueName: \"kubernetes.io/projected/ee548a01-d0b9-4387-a699-041ac6e9b205-kube-api-access-54j5l\") pod \"maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st\" (UID: \"ee548a01-d0b9-4387-a699-041ac6e9b205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" Oct 30 13:00:00.560911 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.560869 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eb0a51c-3364-487b-8aec-77c2cb215dd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb\" (UID: \"9eb0a51c-3364-487b-8aec-77c2cb215dd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" Oct 30 13:00:00.561132 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.561065 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qncwd\" (UniqueName: \"kubernetes.io/projected/de938051-680d-48df-80b7-8e71ee81217b-kube-api-access-qncwd\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8\" (UID: \"de938051-680d-48df-80b7-8e71ee81217b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" Oct 30 13:00:00.561230 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.561216 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjjz\" (UniqueName: \"kubernetes.io/projected/137214d6-09d8-4a4f-bf37-fcef28d37d07-kube-api-access-hcjjz\") pod \"maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw\" (UID: \"137214d6-09d8-4a4f-bf37-fcef28d37d07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" Oct 30 13:00:00.561298 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.561266 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de938051-680d-48df-80b7-8e71ee81217b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8\" (UID: \"de938051-680d-48df-80b7-8e71ee81217b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" Oct 30 13:00:00.561298 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.561292 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee548a01-d0b9-4387-a699-041ac6e9b205-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st\" (UID: \"ee548a01-d0b9-4387-a699-041ac6e9b205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" Oct 30 13:00:00.561415 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.561326 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvl9d\" (UniqueName: \"kubernetes.io/projected/9eb0a51c-3364-487b-8aec-77c2cb215dd7-kube-api-access-qvl9d\") pod \"maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb\" (UID: \"9eb0a51c-3364-487b-8aec-77c2cb215dd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" Oct 30 13:00:00.561415 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.561388 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/137214d6-09d8-4a4f-bf37-fcef28d37d07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw\" (UID: \"137214d6-09d8-4a4f-bf37-fcef28d37d07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" Oct 30 13:00:00.561762 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.561736 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eb0a51c-3364-487b-8aec-77c2cb215dd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb\" (UID: \"9eb0a51c-3364-487b-8aec-77c2cb215dd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" Oct 30 13:00:00.561975 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.561957 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de938051-680d-48df-80b7-8e71ee81217b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8\" (UID: \"de938051-680d-48df-80b7-8e71ee81217b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" Oct 30 13:00:00.562050 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.562032 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee548a01-d0b9-4387-a699-041ac6e9b205-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st\" (UID: \"ee548a01-d0b9-4387-a699-041ac6e9b205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" Oct 30 13:00:00.562139 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.562062 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/137214d6-09d8-4a4f-bf37-fcef28d37d07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw\" (UID: \"137214d6-09d8-4a4f-bf37-fcef28d37d07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" Oct 30 13:00:00.586716 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.586692 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qncwd\" (UniqueName: \"kubernetes.io/projected/de938051-680d-48df-80b7-8e71ee81217b-kube-api-access-qncwd\") pod \"maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8\" (UID: \"de938051-680d-48df-80b7-8e71ee81217b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" Oct 30 13:00:00.586825 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.586801 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54j5l\" (UniqueName: \"kubernetes.io/projected/ee548a01-d0b9-4387-a699-041ac6e9b205-kube-api-access-54j5l\") pod \"maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st\" (UID: \"ee548a01-d0b9-4387-a699-041ac6e9b205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" Oct 30 13:00:00.586971 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.586948 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcjjz\" (UniqueName: \"kubernetes.io/projected/137214d6-09d8-4a4f-bf37-fcef28d37d07-kube-api-access-hcjjz\") pod \"maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw\" (UID: \"137214d6-09d8-4a4f-bf37-fcef28d37d07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" Oct 30 13:00:00.587028 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.587011 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvl9d\" (UniqueName: \"kubernetes.io/projected/9eb0a51c-3364-487b-8aec-77c2cb215dd7-kube-api-access-qvl9d\") pod \"maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb\" (UID: \"9eb0a51c-3364-487b-8aec-77c2cb215dd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" Oct 30 13:00:00.600752 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.600726 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" Oct 30 13:00:00.622676 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.622653 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" Oct 30 13:00:00.658875 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.658833 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" Oct 30 13:00:00.668948 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.668831 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" Oct 30 13:00:00.700368 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.699779 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" Oct 30 13:00:00.707733 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.707527 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" Oct 30 13:00:00.958269 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:00.958233 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4"] Oct 30 13:00:00.958518 ip-10-0-131-76 kubenswrapper[4468]: W1030 13:00:00.958382 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf3ab9f_f6f2_4d98_9f6a_1af9c6b7f778.slice/crio-6de93e6f96804b252f3ba41ee569cf5accbb5ffa2a0a5ae79c69571da854670a WatchSource:0}: Error finding container 6de93e6f96804b252f3ba41ee569cf5accbb5ffa2a0a5ae79c69571da854670a: Status 404 returned error can't find the container with id 6de93e6f96804b252f3ba41ee569cf5accbb5ffa2a0a5ae79c69571da854670a Oct 30 13:00:01.153889 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:01.153859 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq"] Oct 30 13:00:01.201234 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:01.201207 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8"] Oct 30 13:00:01.207063 ip-10-0-131-76 kubenswrapper[4468]: W1030 13:00:01.207028 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3314d4c4_7905_42dd_b342_277da538ba35.slice/crio-bc8faaad0926a7b60c6cb154f809884e0bfae88a8fc3582bc07313ddf4301b0f WatchSource:0}: Error finding container bc8faaad0926a7b60c6cb154f809884e0bfae88a8fc3582bc07313ddf4301b0f: Status 404 returned error can't find the container with id bc8faaad0926a7b60c6cb154f809884e0bfae88a8fc3582bc07313ddf4301b0f Oct 30 13:00:01.207982 ip-10-0-131-76 kubenswrapper[4468]: W1030 13:00:01.207960 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde938051_680d_48df_80b7_8e71ee81217b.slice/crio-57b57ae31b1dd93e0796fdf2945d59ce877a81f44cdaa6676fec5b1c8a527d2d WatchSource:0}: Error finding container 57b57ae31b1dd93e0796fdf2945d59ce877a81f44cdaa6676fec5b1c8a527d2d: Status 404 returned error can't find the container with id 57b57ae31b1dd93e0796fdf2945d59ce877a81f44cdaa6676fec5b1c8a527d2d Oct 30 13:00:01.227494 ip-10-0-131-76 kubenswrapper[4468]: W1030 13:00:01.227462 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137214d6_09d8_4a4f_bf37_fcef28d37d07.slice/crio-e481b04e9f12dae8a1d528da4553d29b0ba628d0c8d5c0b60bb88c76871f4689 WatchSource:0}: Error finding container e481b04e9f12dae8a1d528da4553d29b0ba628d0c8d5c0b60bb88c76871f4689: Status 404 returned error can't find the container with id e481b04e9f12dae8a1d528da4553d29b0ba628d0c8d5c0b60bb88c76871f4689 Oct 30 13:00:01.228451 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:01.228430 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw"] Oct 30 13:00:01.266970 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:01.266945 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb"] Oct 30 13:00:01.267387 ip-10-0-131-76 kubenswrapper[4468]: W1030 13:00:01.267330 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee548a01_d0b9_4387_a699_041ac6e9b205.slice/crio-c6b8f7eb36b9e6a47dacf68c4177ece2b74ee2bf20a385a30b696ad0003056e3 WatchSource:0}: Error finding container c6b8f7eb36b9e6a47dacf68c4177ece2b74ee2bf20a385a30b696ad0003056e3: Status 404 returned error can't find the container with id c6b8f7eb36b9e6a47dacf68c4177ece2b74ee2bf20a385a30b696ad0003056e3 Oct 30 13:00:01.273523 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:01.273501 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st"] Oct 30 13:00:01.274489 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:01.274466 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" event={"ID":"137214d6-09d8-4a4f-bf37-fcef28d37d07","Type":"ContainerStarted","Data":"e481b04e9f12dae8a1d528da4553d29b0ba628d0c8d5c0b60bb88c76871f4689"} Oct 30 13:00:01.275874 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:01.275846 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" event={"ID":"de938051-680d-48df-80b7-8e71ee81217b","Type":"ContainerStarted","Data":"57b57ae31b1dd93e0796fdf2945d59ce877a81f44cdaa6676fec5b1c8a527d2d"} Oct 30 13:00:01.277334 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:01.277310 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" event={"ID":"6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778","Type":"ContainerStarted","Data":"6de93e6f96804b252f3ba41ee569cf5accbb5ffa2a0a5ae79c69571da854670a"} Oct 30 13:00:01.278735 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:01.278711 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" event={"ID":"3314d4c4-7905-42dd-b342-277da538ba35","Type":"ContainerStarted","Data":"bc8faaad0926a7b60c6cb154f809884e0bfae88a8fc3582bc07313ddf4301b0f"} Oct 30 13:00:02.288391 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:02.288297 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" event={"ID":"137214d6-09d8-4a4f-bf37-fcef28d37d07","Type":"ContainerStarted","Data":"17d4c36719306a991d5b0737d9a085e5713cb751a163a83bfe758b21ec7de70c"} Oct 30 13:00:02.290058 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:02.290025 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" event={"ID":"de938051-680d-48df-80b7-8e71ee81217b","Type":"ContainerStarted","Data":"d6407abc628f6b2e22bce75d8623145e64e7be95e6879f08c0f270d2afc45067"} Oct 30 13:00:02.291663 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:02.291633 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" event={"ID":"6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778","Type":"ContainerStarted","Data":"d6be21d37bf5763cd26479d4333e2a6ae674bed7fa8786c9c13d1aa937c8b5fe"} Oct 30 13:00:02.293696 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:02.293670 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" event={"ID":"ee548a01-d0b9-4387-a699-041ac6e9b205","Type":"ContainerStarted","Data":"8b8c8faa189e0b4de926cd3342a172e334911b51704b422b2014ef1a05ec7632"} Oct 30 13:00:02.293824 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:02.293715 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" event={"ID":"ee548a01-d0b9-4387-a699-041ac6e9b205","Type":"ContainerStarted","Data":"c6b8f7eb36b9e6a47dacf68c4177ece2b74ee2bf20a385a30b696ad0003056e3"} Oct 30 13:00:02.295513 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:02.295485 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" event={"ID":"9eb0a51c-3364-487b-8aec-77c2cb215dd7","Type":"ContainerStarted","Data":"7b3a83e8f3cca10417cadc747c562e91b64470ac9938870f359135c65fa3bcd1"} Oct 30 13:00:02.295617 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:02.295530 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" event={"ID":"9eb0a51c-3364-487b-8aec-77c2cb215dd7","Type":"ContainerStarted","Data":"6654f9df9f69bab07b48d7c8bb18d54a95adf07cb0e71ac3f503bd328785116c"} Oct 30 13:00:02.297740 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:02.297712 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" event={"ID":"3314d4c4-7905-42dd-b342-277da538ba35","Type":"ContainerStarted","Data":"714fec0416896b02d53585699869ac2ceb81bb379dd296e39c097858f003d2c9"} Oct 30 13:00:02.343159 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:02.343110 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" podStartSLOduration=2.343058788 podStartE2EDuration="2.343058788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:02.342566017 +0000 UTC m=+3953.376704412" watchObservedRunningTime="2026-10-30 13:00:02.343058788 +0000 UTC m=+3953.377197183" Oct 30 13:00:02.388861 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:02.388810 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" podStartSLOduration=2.388796075 podStartE2EDuration="2.388796075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:02.384958349 +0000 UTC m=+3953.419096746" watchObservedRunningTime="2026-10-30 13:00:02.388796075 +0000 UTC m=+3953.422934471" Oct 30 13:00:02.433548 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:02.433485 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" podStartSLOduration=2.433470976 podStartE2EDuration="2.433470976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:02.4272466 +0000 UTC m=+3953.461384995" watchObservedRunningTime="2026-10-30 13:00:02.433470976 +0000 UTC m=+3953.467609420" Oct 30 13:00:02.471064 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:02.471016 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" podStartSLOduration=2.471002432 podStartE2EDuration="2.471002432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:02.470192066 +0000 UTC m=+3953.504330463" watchObservedRunningTime="2026-10-30 13:00:02.471002432 +0000 UTC m=+3953.505140827" Oct 30 13:00:02.514196 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:02.514138 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" podStartSLOduration=2.5141198080000002 podStartE2EDuration="2.514119808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:02.512347588 +0000 UTC m=+3953.546485985" watchObservedRunningTime="2026-10-30 13:00:02.514119808 +0000 UTC m=+3953.548258207" Oct 30 13:00:02.557762 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:02.557684 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" podStartSLOduration=2.557669197 podStartE2EDuration="2.557669197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:02.555016915 +0000 UTC m=+3953.589155312" watchObservedRunningTime="2026-10-30 13:00:02.557669197 +0000 UTC m=+3953.591807633" Oct 30 13:00:03.308418 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:03.308215 4468 generic.go:356] "Generic (PLEG): container finished" podID="9eb0a51c-3364-487b-8aec-77c2cb215dd7" containerID="7b3a83e8f3cca10417cadc747c562e91b64470ac9938870f359135c65fa3bcd1" exitCode=60 Oct 30 13:00:03.308897 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:03.308541 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" event={"ID":"9eb0a51c-3364-487b-8aec-77c2cb215dd7","Type":"ContainerDied","Data":"7b3a83e8f3cca10417cadc747c562e91b64470ac9938870f359135c65fa3bcd1"} Oct 30 13:00:03.308897 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:03.308731 4468 scope.go:122] "RemoveContainer" containerID="7b3a83e8f3cca10417cadc747c562e91b64470ac9938870f359135c65fa3bcd1" Oct 30 13:00:03.310413 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:03.310389 4468 generic.go:356] "Generic (PLEG): container finished" podID="3314d4c4-7905-42dd-b342-277da538ba35" containerID="714fec0416896b02d53585699869ac2ceb81bb379dd296e39c097858f003d2c9" exitCode=60 Oct 30 13:00:03.310525 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:03.310440 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" event={"ID":"3314d4c4-7905-42dd-b342-277da538ba35","Type":"ContainerDied","Data":"714fec0416896b02d53585699869ac2ceb81bb379dd296e39c097858f003d2c9"} Oct 30 13:00:03.310703 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:03.310678 4468 scope.go:122] "RemoveContainer" containerID="714fec0416896b02d53585699869ac2ceb81bb379dd296e39c097858f003d2c9" Oct 30 13:00:03.312486 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:03.312458 4468 generic.go:356] "Generic (PLEG): container finished" podID="137214d6-09d8-4a4f-bf37-fcef28d37d07" containerID="17d4c36719306a991d5b0737d9a085e5713cb751a163a83bfe758b21ec7de70c" exitCode=60 Oct 30 13:00:03.312559 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:03.312520 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" event={"ID":"137214d6-09d8-4a4f-bf37-fcef28d37d07","Type":"ContainerDied","Data":"17d4c36719306a991d5b0737d9a085e5713cb751a163a83bfe758b21ec7de70c"} Oct 30 13:00:03.312785 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:03.312770 4468 scope.go:122] "RemoveContainer" containerID="17d4c36719306a991d5b0737d9a085e5713cb751a163a83bfe758b21ec7de70c" Oct 30 13:00:03.314606 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:03.314567 4468 generic.go:356] "Generic (PLEG): container finished" podID="de938051-680d-48df-80b7-8e71ee81217b" containerID="d6407abc628f6b2e22bce75d8623145e64e7be95e6879f08c0f270d2afc45067" exitCode=60 Oct 30 13:00:03.314671 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:03.314637 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" event={"ID":"de938051-680d-48df-80b7-8e71ee81217b","Type":"ContainerDied","Data":"d6407abc628f6b2e22bce75d8623145e64e7be95e6879f08c0f270d2afc45067"} Oct 30 13:00:03.314859 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:03.314844 4468 scope.go:122] "RemoveContainer" containerID="d6407abc628f6b2e22bce75d8623145e64e7be95e6879f08c0f270d2afc45067" Oct 30 13:00:03.316362 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:03.316341 4468 generic.go:356] "Generic (PLEG): container finished" podID="6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778" containerID="d6be21d37bf5763cd26479d4333e2a6ae674bed7fa8786c9c13d1aa937c8b5fe" exitCode=60 Oct 30 13:00:03.316446 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:03.316417 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" event={"ID":"6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778","Type":"ContainerDied","Data":"d6be21d37bf5763cd26479d4333e2a6ae674bed7fa8786c9c13d1aa937c8b5fe"} Oct 30 13:00:03.316623 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:03.316608 4468 scope.go:122] "RemoveContainer" containerID="d6be21d37bf5763cd26479d4333e2a6ae674bed7fa8786c9c13d1aa937c8b5fe" Oct 30 13:00:03.318126 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:03.318102 4468 generic.go:356] "Generic (PLEG): container finished" podID="ee548a01-d0b9-4387-a699-041ac6e9b205" containerID="8b8c8faa189e0b4de926cd3342a172e334911b51704b422b2014ef1a05ec7632" exitCode=60 Oct 30 13:00:03.318215 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:03.318144 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" event={"ID":"ee548a01-d0b9-4387-a699-041ac6e9b205","Type":"ContainerDied","Data":"8b8c8faa189e0b4de926cd3342a172e334911b51704b422b2014ef1a05ec7632"} Oct 30 13:00:03.318391 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:03.318375 4468 scope.go:122] "RemoveContainer" containerID="8b8c8faa189e0b4de926cd3342a172e334911b51704b422b2014ef1a05ec7632" Oct 30 13:00:04.339593 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:04.334873 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" event={"ID":"9eb0a51c-3364-487b-8aec-77c2cb215dd7","Type":"ContainerStarted","Data":"3039f29078a85a9067c4691a319dbeb75d961ecd99c5398e42a866838b203004"} Oct 30 13:00:04.339593 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:04.335437 4468 scope.go:122] "RemoveContainer" containerID="3039f29078a85a9067c4691a319dbeb75d961ecd99c5398e42a866838b203004" Oct 30 13:00:04.339593 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:04.335874 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb_odh-ai-gateway-infra(9eb0a51c-3364-487b-8aec-77c2cb215dd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" podUID="9eb0a51c-3364-487b-8aec-77c2cb215dd7" Oct 30 13:00:04.339593 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:04.338842 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" event={"ID":"3314d4c4-7905-42dd-b342-277da538ba35","Type":"ContainerStarted","Data":"bb13dc3e24d79b5bd98e2174850de72f6301dffbff564886bb125e5726a0ac83"} Oct 30 13:00:04.341982 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:04.341356 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" event={"ID":"137214d6-09d8-4a4f-bf37-fcef28d37d07","Type":"ContainerStarted","Data":"84388cdc3d3ec21e12655f2e21af6d06660e8761e0ddf2cf33acd0d26a5f16ef"} Oct 30 13:00:04.344055 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:04.344034 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" event={"ID":"de938051-680d-48df-80b7-8e71ee81217b","Type":"ContainerStarted","Data":"6e0791f9d51faf196bfe73202288391211835938a683a247aa63c6010f522c0d"} Oct 30 13:00:04.345995 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:04.345970 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" event={"ID":"6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778","Type":"ContainerStarted","Data":"9eb1fd1aca545712cb2f6217e786aa8f40d72ca9eaff4066feab27202d88cd0f"} Oct 30 13:00:04.347958 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:04.347936 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" event={"ID":"ee548a01-d0b9-4387-a699-041ac6e9b205","Type":"ContainerStarted","Data":"dcfedc3c0e2474223597f014943617e3c43a9b31af42a2fe73599b1c35117069"} Oct 30 13:00:05.360654 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.360623 4468 generic.go:356] "Generic (PLEG): container finished" podID="ee548a01-d0b9-4387-a699-041ac6e9b205" containerID="dcfedc3c0e2474223597f014943617e3c43a9b31af42a2fe73599b1c35117069" exitCode=60 Oct 30 13:00:05.361137 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.360709 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" event={"ID":"ee548a01-d0b9-4387-a699-041ac6e9b205","Type":"ContainerDied","Data":"dcfedc3c0e2474223597f014943617e3c43a9b31af42a2fe73599b1c35117069"} Oct 30 13:00:05.361137 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.360790 4468 scope.go:122] "RemoveContainer" containerID="8b8c8faa189e0b4de926cd3342a172e334911b51704b422b2014ef1a05ec7632" Oct 30 13:00:05.361137 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.361013 4468 scope.go:122] "RemoveContainer" containerID="dcfedc3c0e2474223597f014943617e3c43a9b31af42a2fe73599b1c35117069" Oct 30 13:00:05.361402 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:05.361377 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st_odh-ai-gateway-infra(ee548a01-d0b9-4387-a699-041ac6e9b205)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" podUID="ee548a01-d0b9-4387-a699-041ac6e9b205" Oct 30 13:00:05.363221 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.363200 4468 generic.go:356] "Generic (PLEG): container finished" podID="9eb0a51c-3364-487b-8aec-77c2cb215dd7" containerID="3039f29078a85a9067c4691a319dbeb75d961ecd99c5398e42a866838b203004" exitCode=60 Oct 30 13:00:05.363340 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.363257 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" event={"ID":"9eb0a51c-3364-487b-8aec-77c2cb215dd7","Type":"ContainerDied","Data":"3039f29078a85a9067c4691a319dbeb75d961ecd99c5398e42a866838b203004"} Oct 30 13:00:05.363555 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.363536 4468 scope.go:122] "RemoveContainer" containerID="3039f29078a85a9067c4691a319dbeb75d961ecd99c5398e42a866838b203004" Oct 30 13:00:05.363791 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:05.363771 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb_odh-ai-gateway-infra(9eb0a51c-3364-487b-8aec-77c2cb215dd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" podUID="9eb0a51c-3364-487b-8aec-77c2cb215dd7" Oct 30 13:00:05.365120 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.365099 4468 generic.go:356] "Generic (PLEG): container finished" podID="3314d4c4-7905-42dd-b342-277da538ba35" containerID="bb13dc3e24d79b5bd98e2174850de72f6301dffbff564886bb125e5726a0ac83" exitCode=60 Oct 30 13:00:05.365228 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.365178 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" event={"ID":"3314d4c4-7905-42dd-b342-277da538ba35","Type":"ContainerDied","Data":"bb13dc3e24d79b5bd98e2174850de72f6301dffbff564886bb125e5726a0ac83"} Oct 30 13:00:05.365398 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.365382 4468 scope.go:122] "RemoveContainer" containerID="bb13dc3e24d79b5bd98e2174850de72f6301dffbff564886bb125e5726a0ac83" Oct 30 13:00:05.365650 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:05.365621 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq_odh-ai-gateway-infra(3314d4c4-7905-42dd-b342-277da538ba35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" podUID="3314d4c4-7905-42dd-b342-277da538ba35" Oct 30 13:00:05.367104 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.367059 4468 generic.go:356] "Generic (PLEG): container finished" podID="137214d6-09d8-4a4f-bf37-fcef28d37d07" containerID="84388cdc3d3ec21e12655f2e21af6d06660e8761e0ddf2cf33acd0d26a5f16ef" exitCode=60 Oct 30 13:00:05.367188 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.367106 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" event={"ID":"137214d6-09d8-4a4f-bf37-fcef28d37d07","Type":"ContainerDied","Data":"84388cdc3d3ec21e12655f2e21af6d06660e8761e0ddf2cf33acd0d26a5f16ef"} Oct 30 13:00:05.367337 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.367324 4468 scope.go:122] "RemoveContainer" containerID="84388cdc3d3ec21e12655f2e21af6d06660e8761e0ddf2cf33acd0d26a5f16ef" Oct 30 13:00:05.367566 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:05.367536 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw_odh-ai-gateway-infra(137214d6-09d8-4a4f-bf37-fcef28d37d07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" podUID="137214d6-09d8-4a4f-bf37-fcef28d37d07" Oct 30 13:00:05.368989 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.368970 4468 generic.go:356] "Generic (PLEG): container finished" podID="de938051-680d-48df-80b7-8e71ee81217b" containerID="6e0791f9d51faf196bfe73202288391211835938a683a247aa63c6010f522c0d" exitCode=60 Oct 30 13:00:05.369117 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.369049 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" event={"ID":"de938051-680d-48df-80b7-8e71ee81217b","Type":"ContainerDied","Data":"6e0791f9d51faf196bfe73202288391211835938a683a247aa63c6010f522c0d"} Oct 30 13:00:05.369287 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.369268 4468 scope.go:122] "RemoveContainer" containerID="6e0791f9d51faf196bfe73202288391211835938a683a247aa63c6010f522c0d" Oct 30 13:00:05.369545 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:05.369515 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8_odh-ai-gateway-infra(de938051-680d-48df-80b7-8e71ee81217b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" podUID="de938051-680d-48df-80b7-8e71ee81217b" Oct 30 13:00:05.371310 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.371290 4468 generic.go:356] "Generic (PLEG): container finished" podID="6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778" containerID="9eb1fd1aca545712cb2f6217e786aa8f40d72ca9eaff4066feab27202d88cd0f" exitCode=60 Oct 30 13:00:05.371479 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.371331 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" event={"ID":"6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778","Type":"ContainerDied","Data":"9eb1fd1aca545712cb2f6217e786aa8f40d72ca9eaff4066feab27202d88cd0f"} Oct 30 13:00:05.371657 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.371642 4468 scope.go:122] "RemoveContainer" containerID="9eb1fd1aca545712cb2f6217e786aa8f40d72ca9eaff4066feab27202d88cd0f" Oct 30 13:00:05.371850 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:05.371834 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4_odh-ai-gateway-infra(6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" podUID="6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778" Oct 30 13:00:05.376346 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.376327 4468 scope.go:122] "RemoveContainer" containerID="7b3a83e8f3cca10417cadc747c562e91b64470ac9938870f359135c65fa3bcd1" Oct 30 13:00:05.391214 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.391196 4468 scope.go:122] "RemoveContainer" containerID="714fec0416896b02d53585699869ac2ceb81bb379dd296e39c097858f003d2c9" Oct 30 13:00:05.408615 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.408594 4468 scope.go:122] "RemoveContainer" containerID="17d4c36719306a991d5b0737d9a085e5713cb751a163a83bfe758b21ec7de70c" Oct 30 13:00:05.424011 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.423993 4468 scope.go:122] "RemoveContainer" containerID="d6407abc628f6b2e22bce75d8623145e64e7be95e6879f08c0f270d2afc45067" Oct 30 13:00:05.437898 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:05.437883 4468 scope.go:122] "RemoveContainer" containerID="d6be21d37bf5763cd26479d4333e2a6ae674bed7fa8786c9c13d1aa937c8b5fe" Oct 30 13:00:06.381511 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:06.381481 4468 scope.go:122] "RemoveContainer" containerID="dcfedc3c0e2474223597f014943617e3c43a9b31af42a2fe73599b1c35117069" Oct 30 13:00:06.382111 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:06.381857 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st_odh-ai-gateway-infra(ee548a01-d0b9-4387-a699-041ac6e9b205)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" podUID="ee548a01-d0b9-4387-a699-041ac6e9b205" Oct 30 13:00:06.383345 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:06.383323 4468 scope.go:122] "RemoveContainer" containerID="3039f29078a85a9067c4691a319dbeb75d961ecd99c5398e42a866838b203004" Oct 30 13:00:06.383720 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:06.383694 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb_odh-ai-gateway-infra(9eb0a51c-3364-487b-8aec-77c2cb215dd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" podUID="9eb0a51c-3364-487b-8aec-77c2cb215dd7" Oct 30 13:00:06.384886 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:06.384867 4468 scope.go:122] "RemoveContainer" containerID="bb13dc3e24d79b5bd98e2174850de72f6301dffbff564886bb125e5726a0ac83" Oct 30 13:00:06.385178 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:06.385156 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq_odh-ai-gateway-infra(3314d4c4-7905-42dd-b342-277da538ba35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" podUID="3314d4c4-7905-42dd-b342-277da538ba35" Oct 30 13:00:06.386820 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:06.386790 4468 scope.go:122] "RemoveContainer" containerID="84388cdc3d3ec21e12655f2e21af6d06660e8761e0ddf2cf33acd0d26a5f16ef" Oct 30 13:00:06.387092 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:06.387051 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw_odh-ai-gateway-infra(137214d6-09d8-4a4f-bf37-fcef28d37d07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" podUID="137214d6-09d8-4a4f-bf37-fcef28d37d07" Oct 30 13:00:06.388366 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:06.388350 4468 scope.go:122] "RemoveContainer" containerID="6e0791f9d51faf196bfe73202288391211835938a683a247aa63c6010f522c0d" Oct 30 13:00:06.388607 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:06.388588 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8_odh-ai-gateway-infra(de938051-680d-48df-80b7-8e71ee81217b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" podUID="de938051-680d-48df-80b7-8e71ee81217b" Oct 30 13:00:06.389919 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:06.389901 4468 scope.go:122] "RemoveContainer" containerID="9eb1fd1aca545712cb2f6217e786aa8f40d72ca9eaff4066feab27202d88cd0f" Oct 30 13:00:06.390174 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:06.390151 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4_odh-ai-gateway-infra(6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" podUID="6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778" Oct 30 13:00:16.082302 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:16.082267 4468 scope.go:122] "RemoveContainer" containerID="3039f29078a85a9067c4691a319dbeb75d961ecd99c5398e42a866838b203004" Oct 30 13:00:17.082547 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:17.082515 4468 scope.go:122] "RemoveContainer" containerID="dcfedc3c0e2474223597f014943617e3c43a9b31af42a2fe73599b1c35117069" Oct 30 13:00:17.082938 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:17.082652 4468 scope.go:122] "RemoveContainer" containerID="84388cdc3d3ec21e12655f2e21af6d06660e8761e0ddf2cf33acd0d26a5f16ef" Oct 30 13:00:17.082938 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:17.082776 4468 scope.go:122] "RemoveContainer" containerID="bb13dc3e24d79b5bd98e2174850de72f6301dffbff564886bb125e5726a0ac83" Oct 30 13:00:17.498120 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:17.498064 4468 generic.go:356] "Generic (PLEG): container finished" podID="9eb0a51c-3364-487b-8aec-77c2cb215dd7" containerID="a296b948169da89d4f800067a39f5025a0aaf29ab1112c9e408b30bcd4f7d404" exitCode=60 Oct 30 13:00:17.498120 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:17.498100 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" event={"ID":"9eb0a51c-3364-487b-8aec-77c2cb215dd7","Type":"ContainerDied","Data":"a296b948169da89d4f800067a39f5025a0aaf29ab1112c9e408b30bcd4f7d404"} Oct 30 13:00:17.498375 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:17.498171 4468 scope.go:122] "RemoveContainer" containerID="3039f29078a85a9067c4691a319dbeb75d961ecd99c5398e42a866838b203004" Oct 30 13:00:17.498592 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:17.498576 4468 scope.go:122] "RemoveContainer" containerID="a296b948169da89d4f800067a39f5025a0aaf29ab1112c9e408b30bcd4f7d404" Oct 30 13:00:17.498961 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:17.498938 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb_odh-ai-gateway-infra(9eb0a51c-3364-487b-8aec-77c2cb215dd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" podUID="9eb0a51c-3364-487b-8aec-77c2cb215dd7" Oct 30 13:00:18.083117 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.083053 4468 scope.go:122] "RemoveContainer" containerID="6e0791f9d51faf196bfe73202288391211835938a683a247aa63c6010f522c0d" Oct 30 13:00:18.519853 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.519824 4468 generic.go:356] "Generic (PLEG): container finished" podID="137214d6-09d8-4a4f-bf37-fcef28d37d07" containerID="093327a334a814b2865fff02f4ced9b1b459be084bfa3235fd0db56bc961cff3" exitCode=60 Oct 30 13:00:18.519988 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.519893 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" event={"ID":"137214d6-09d8-4a4f-bf37-fcef28d37d07","Type":"ContainerDied","Data":"093327a334a814b2865fff02f4ced9b1b459be084bfa3235fd0db56bc961cff3"} Oct 30 13:00:18.519988 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.519953 4468 scope.go:122] "RemoveContainer" containerID="84388cdc3d3ec21e12655f2e21af6d06660e8761e0ddf2cf33acd0d26a5f16ef" Oct 30 13:00:18.520418 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.520404 4468 scope.go:122] "RemoveContainer" containerID="093327a334a814b2865fff02f4ced9b1b459be084bfa3235fd0db56bc961cff3" Oct 30 13:00:18.520786 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:18.520766 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw_odh-ai-gateway-infra(137214d6-09d8-4a4f-bf37-fcef28d37d07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" podUID="137214d6-09d8-4a4f-bf37-fcef28d37d07" Oct 30 13:00:18.522642 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.522394 4468 generic.go:356] "Generic (PLEG): container finished" podID="ee548a01-d0b9-4387-a699-041ac6e9b205" containerID="1e42939e023c522142dbc430dba23e2f417cd73595d08d59096f699f556ed3f6" exitCode=60 Oct 30 13:00:18.522642 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.522489 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" event={"ID":"ee548a01-d0b9-4387-a699-041ac6e9b205","Type":"ContainerDied","Data":"1e42939e023c522142dbc430dba23e2f417cd73595d08d59096f699f556ed3f6"} Oct 30 13:00:18.522969 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.522949 4468 scope.go:122] "RemoveContainer" containerID="1e42939e023c522142dbc430dba23e2f417cd73595d08d59096f699f556ed3f6" Oct 30 13:00:18.523419 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:18.523397 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st_odh-ai-gateway-infra(ee548a01-d0b9-4387-a699-041ac6e9b205)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" podUID="ee548a01-d0b9-4387-a699-041ac6e9b205" Oct 30 13:00:18.526407 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.526387 4468 generic.go:356] "Generic (PLEG): container finished" podID="3314d4c4-7905-42dd-b342-277da538ba35" containerID="3317f76b8376139a82eeafef296af7fd5231aca17c622ace5e461f1dd9d901fb" exitCode=60 Oct 30 13:00:18.526505 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.526415 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" event={"ID":"3314d4c4-7905-42dd-b342-277da538ba35","Type":"ContainerDied","Data":"3317f76b8376139a82eeafef296af7fd5231aca17c622ace5e461f1dd9d901fb"} Oct 30 13:00:18.526896 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.526879 4468 scope.go:122] "RemoveContainer" containerID="3317f76b8376139a82eeafef296af7fd5231aca17c622ace5e461f1dd9d901fb" Oct 30 13:00:18.527329 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:18.527307 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq_odh-ai-gateway-infra(3314d4c4-7905-42dd-b342-277da538ba35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" podUID="3314d4c4-7905-42dd-b342-277da538ba35" Oct 30 13:00:18.534091 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.534059 4468 scope.go:122] "RemoveContainer" containerID="dcfedc3c0e2474223597f014943617e3c43a9b31af42a2fe73599b1c35117069" Oct 30 13:00:18.547982 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.547957 4468 scope.go:122] "RemoveContainer" containerID="bb13dc3e24d79b5bd98e2174850de72f6301dffbff564886bb125e5726a0ac83" Oct 30 13:00:18.569153 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.568553 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb"] Oct 30 13:00:18.772342 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.772317 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" Oct 30 13:00:18.884161 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.884126 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eb0a51c-3364-487b-8aec-77c2cb215dd7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eb0a51c-3364-487b-8aec-77c2cb215dd7-maas-api-service-ca\") pod \"9eb0a51c-3364-487b-8aec-77c2cb215dd7\" (UID: \"9eb0a51c-3364-487b-8aec-77c2cb215dd7\") " Oct 30 13:00:18.884501 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.884484 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb0a51c-3364-487b-8aec-77c2cb215dd7-kube-api-access-qvl9d\" (UniqueName: \"kubernetes.io/projected/9eb0a51c-3364-487b-8aec-77c2cb215dd7-kube-api-access-qvl9d\") pod \"9eb0a51c-3364-487b-8aec-77c2cb215dd7\" (UID: \"9eb0a51c-3364-487b-8aec-77c2cb215dd7\") " Oct 30 13:00:18.884593 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.884510 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb0a51c-3364-487b-8aec-77c2cb215dd7-maas-api-service-ca" pod "9eb0a51c-3364-487b-8aec-77c2cb215dd7" (UID: "9eb0a51c-3364-487b-8aec-77c2cb215dd7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:18.884933 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.884901 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eb0a51c-3364-487b-8aec-77c2cb215dd7-maas-api-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:00:18.886818 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.886795 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb0a51c-3364-487b-8aec-77c2cb215dd7-kube-api-access-qvl9d" pod "9eb0a51c-3364-487b-8aec-77c2cb215dd7" (UID: "9eb0a51c-3364-487b-8aec-77c2cb215dd7"). InnerVolumeSpecName "kube-api-access-qvl9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:18.986270 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:18.986195 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvl9d\" (UniqueName: \"kubernetes.io/projected/9eb0a51c-3364-487b-8aec-77c2cb215dd7-kube-api-access-qvl9d\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:00:19.088238 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.088208 4468 scope.go:122] "RemoveContainer" containerID="9eb1fd1aca545712cb2f6217e786aa8f40d72ca9eaff4066feab27202d88cd0f" Oct 30 13:00:19.537247 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.537215 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" Oct 30 13:00:19.537436 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.537250 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb" event={"ID":"9eb0a51c-3364-487b-8aec-77c2cb215dd7","Type":"ContainerDied","Data":"6654f9df9f69bab07b48d7c8bb18d54a95adf07cb0e71ac3f503bd328785116c"} Oct 30 13:00:19.537436 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.537313 4468 scope.go:122] "RemoveContainer" containerID="a296b948169da89d4f800067a39f5025a0aaf29ab1112c9e408b30bcd4f7d404" Oct 30 13:00:19.543224 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.543201 4468 generic.go:356] "Generic (PLEG): container finished" podID="de938051-680d-48df-80b7-8e71ee81217b" containerID="e9df172fda691cbaa94134661381ba4d50274d3c194a24a76e6fb7c9536fceff" exitCode=60 Oct 30 13:00:19.543348 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.543275 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" event={"ID":"de938051-680d-48df-80b7-8e71ee81217b","Type":"ContainerDied","Data":"e9df172fda691cbaa94134661381ba4d50274d3c194a24a76e6fb7c9536fceff"} Oct 30 13:00:19.543723 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.543705 4468 scope.go:122] "RemoveContainer" containerID="e9df172fda691cbaa94134661381ba4d50274d3c194a24a76e6fb7c9536fceff" Oct 30 13:00:19.544097 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:19.544057 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8_odh-ai-gateway-infra(de938051-680d-48df-80b7-8e71ee81217b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" podUID="de938051-680d-48df-80b7-8e71ee81217b" Oct 30 13:00:19.549704 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.549687 4468 scope.go:122] "RemoveContainer" containerID="6e0791f9d51faf196bfe73202288391211835938a683a247aa63c6010f522c0d" Oct 30 13:00:19.585600 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.585538 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw"] Oct 30 13:00:19.619424 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.619400 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb"] Oct 30 13:00:19.628686 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.628655 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c3f12-29889420-5c8zb"] Oct 30 13:00:19.703265 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.703234 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq"] Oct 30 13:00:19.747850 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.747808 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st"] Oct 30 13:00:19.809929 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.809571 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" Oct 30 13:00:19.901168 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.901136 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/137214d6-09d8-4a4f-bf37-fcef28d37d07-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/137214d6-09d8-4a4f-bf37-fcef28d37d07-maas-api-service-ca\") pod \"137214d6-09d8-4a4f-bf37-fcef28d37d07\" (UID: \"137214d6-09d8-4a4f-bf37-fcef28d37d07\") " Oct 30 13:00:19.901307 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.901235 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137214d6-09d8-4a4f-bf37-fcef28d37d07-kube-api-access-hcjjz\" (UniqueName: \"kubernetes.io/projected/137214d6-09d8-4a4f-bf37-fcef28d37d07-kube-api-access-hcjjz\") pod \"137214d6-09d8-4a4f-bf37-fcef28d37d07\" (UID: \"137214d6-09d8-4a4f-bf37-fcef28d37d07\") " Oct 30 13:00:19.904097 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.901517 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137214d6-09d8-4a4f-bf37-fcef28d37d07-maas-api-service-ca" pod "137214d6-09d8-4a4f-bf37-fcef28d37d07" (UID: "137214d6-09d8-4a4f-bf37-fcef28d37d07"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:19.904097 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.902048 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/137214d6-09d8-4a4f-bf37-fcef28d37d07-maas-api-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:00:19.911664 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.911631 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137214d6-09d8-4a4f-bf37-fcef28d37d07-kube-api-access-hcjjz" pod "137214d6-09d8-4a4f-bf37-fcef28d37d07" (UID: "137214d6-09d8-4a4f-bf37-fcef28d37d07"). InnerVolumeSpecName "kube-api-access-hcjjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:19.920083 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.920036 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" Oct 30 13:00:19.938664 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:19.938631 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" Oct 30 13:00:20.006143 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.003271 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee548a01-d0b9-4387-a699-041ac6e9b205-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee548a01-d0b9-4387-a699-041ac6e9b205-maas-api-service-ca\") pod \"ee548a01-d0b9-4387-a699-041ac6e9b205\" (UID: \"ee548a01-d0b9-4387-a699-041ac6e9b205\") " Oct 30 13:00:20.006143 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.003404 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee548a01-d0b9-4387-a699-041ac6e9b205-kube-api-access-54j5l\" (UniqueName: \"kubernetes.io/projected/ee548a01-d0b9-4387-a699-041ac6e9b205-kube-api-access-54j5l\") pod \"ee548a01-d0b9-4387-a699-041ac6e9b205\" (UID: \"ee548a01-d0b9-4387-a699-041ac6e9b205\") " Oct 30 13:00:20.006143 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.003807 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee548a01-d0b9-4387-a699-041ac6e9b205-maas-api-service-ca" pod "ee548a01-d0b9-4387-a699-041ac6e9b205" (UID: "ee548a01-d0b9-4387-a699-041ac6e9b205"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:20.006847 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.006786 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee548a01-d0b9-4387-a699-041ac6e9b205-kube-api-access-54j5l" pod "ee548a01-d0b9-4387-a699-041ac6e9b205" (UID: "ee548a01-d0b9-4387-a699-041ac6e9b205"). InnerVolumeSpecName "kube-api-access-54j5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:20.016260 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.016205 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3314d4c4-7905-42dd-b342-277da538ba35-kube-api-access-s7rqx\" (UniqueName: \"kubernetes.io/projected/3314d4c4-7905-42dd-b342-277da538ba35-kube-api-access-s7rqx\") pod \"3314d4c4-7905-42dd-b342-277da538ba35\" (UID: \"3314d4c4-7905-42dd-b342-277da538ba35\") " Oct 30 13:00:20.016260 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.016265 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3314d4c4-7905-42dd-b342-277da538ba35-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3314d4c4-7905-42dd-b342-277da538ba35-maas-api-service-ca\") pod \"3314d4c4-7905-42dd-b342-277da538ba35\" (UID: \"3314d4c4-7905-42dd-b342-277da538ba35\") " Oct 30 13:00:20.017168 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.017130 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3314d4c4-7905-42dd-b342-277da538ba35-maas-api-service-ca" pod "3314d4c4-7905-42dd-b342-277da538ba35" (UID: "3314d4c4-7905-42dd-b342-277da538ba35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:20.017168 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.017168 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee548a01-d0b9-4387-a699-041ac6e9b205-maas-api-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:00:20.017402 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.017189 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54j5l\" (UniqueName: \"kubernetes.io/projected/ee548a01-d0b9-4387-a699-041ac6e9b205-kube-api-access-54j5l\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:00:20.017402 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.017204 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcjjz\" (UniqueName: \"kubernetes.io/projected/137214d6-09d8-4a4f-bf37-fcef28d37d07-kube-api-access-hcjjz\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:00:20.019809 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.019770 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3314d4c4-7905-42dd-b342-277da538ba35-kube-api-access-s7rqx" pod "3314d4c4-7905-42dd-b342-277da538ba35" (UID: "3314d4c4-7905-42dd-b342-277da538ba35"). InnerVolumeSpecName "kube-api-access-s7rqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:20.120150 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.120052 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7rqx\" (UniqueName: \"kubernetes.io/projected/3314d4c4-7905-42dd-b342-277da538ba35-kube-api-access-s7rqx\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:00:20.120150 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.120107 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3314d4c4-7905-42dd-b342-277da538ba35-maas-api-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:00:20.557001 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.556971 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" Oct 30 13:00:20.557237 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.557015 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq" event={"ID":"3314d4c4-7905-42dd-b342-277da538ba35","Type":"ContainerDied","Data":"bc8faaad0926a7b60c6cb154f809884e0bfae88a8fc3582bc07313ddf4301b0f"} Oct 30 13:00:20.557237 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.557116 4468 scope.go:122] "RemoveContainer" containerID="3317f76b8376139a82eeafef296af7fd5231aca17c622ace5e461f1dd9d901fb" Oct 30 13:00:20.558596 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.558577 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" event={"ID":"137214d6-09d8-4a4f-bf37-fcef28d37d07","Type":"ContainerDied","Data":"e481b04e9f12dae8a1d528da4553d29b0ba628d0c8d5c0b60bb88c76871f4689"} Oct 30 13:00:20.558703 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.558581 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw" Oct 30 13:00:20.562908 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.562881 4468 generic.go:356] "Generic (PLEG): container finished" podID="6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778" containerID="49a90f68f4e5724be7c72efea200ee481249d2f565a53efc31e6649c6bdc399f" exitCode=60 Oct 30 13:00:20.563039 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.562938 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" event={"ID":"6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778","Type":"ContainerDied","Data":"49a90f68f4e5724be7c72efea200ee481249d2f565a53efc31e6649c6bdc399f"} Oct 30 13:00:20.563466 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.563445 4468 scope.go:122] "RemoveContainer" containerID="49a90f68f4e5724be7c72efea200ee481249d2f565a53efc31e6649c6bdc399f" Oct 30 13:00:20.563788 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:20.563767 4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4_odh-ai-gateway-infra(6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" podUID="6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778" Oct 30 13:00:20.564664 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.564646 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" Oct 30 13:00:20.564664 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.564655 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st" event={"ID":"ee548a01-d0b9-4387-a699-041ac6e9b205","Type":"ContainerDied","Data":"c6b8f7eb36b9e6a47dacf68c4177ece2b74ee2bf20a385a30b696ad0003056e3"} Oct 30 13:00:20.575283 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.573126 4468 scope.go:122] "RemoveContainer" containerID="093327a334a814b2865fff02f4ced9b1b459be084bfa3235fd0db56bc961cff3" Oct 30 13:00:20.586481 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.586458 4468 scope.go:122] "RemoveContainer" containerID="9eb1fd1aca545712cb2f6217e786aa8f40d72ca9eaff4066feab27202d88cd0f" Oct 30 13:00:20.600656 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.600635 4468 scope.go:122] "RemoveContainer" containerID="1e42939e023c522142dbc430dba23e2f417cd73595d08d59096f699f556ed3f6" Oct 30 13:00:20.648761 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.648730 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw"] Oct 30 13:00:20.677131 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.677098 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-734ffe-29889420-qdktw"] Oct 30 13:00:20.696894 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.696862 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8"] Oct 30 13:00:20.809820 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.809730 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq"] Oct 30 13:00:20.822333 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.822296 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f920f7-29889420-wgqqq"] Oct 30 13:00:20.890941 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.890910 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st"] Oct 30 13:00:20.895284 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.895260 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" Oct 30 13:00:20.904663 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.904637 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-eab799-29889420-qq2st"] Oct 30 13:00:20.936559 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.936524 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de938051-680d-48df-80b7-8e71ee81217b-kube-api-access-qncwd\" (UniqueName: \"kubernetes.io/projected/de938051-680d-48df-80b7-8e71ee81217b-kube-api-access-qncwd\") pod \"de938051-680d-48df-80b7-8e71ee81217b\" (UID: \"de938051-680d-48df-80b7-8e71ee81217b\") " Oct 30 13:00:20.936751 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.936736 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de938051-680d-48df-80b7-8e71ee81217b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de938051-680d-48df-80b7-8e71ee81217b-maas-api-service-ca\") pod \"de938051-680d-48df-80b7-8e71ee81217b\" (UID: \"de938051-680d-48df-80b7-8e71ee81217b\") " Oct 30 13:00:20.937168 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.937140 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de938051-680d-48df-80b7-8e71ee81217b-maas-api-service-ca" pod "de938051-680d-48df-80b7-8e71ee81217b" (UID: "de938051-680d-48df-80b7-8e71ee81217b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:20.937456 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.937438 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de938051-680d-48df-80b7-8e71ee81217b-maas-api-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:00:20.938746 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:20.938721 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de938051-680d-48df-80b7-8e71ee81217b-kube-api-access-qncwd" pod "de938051-680d-48df-80b7-8e71ee81217b" (UID: "de938051-680d-48df-80b7-8e71ee81217b"). InnerVolumeSpecName "kube-api-access-qncwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:21.038682 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:21.038637 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qncwd\" (UniqueName: \"kubernetes.io/projected/de938051-680d-48df-80b7-8e71ee81217b-kube-api-access-qncwd\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:00:21.092927 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:21.092839 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="137214d6-09d8-4a4f-bf37-fcef28d37d07" path="/var/lib/kubelet/pods/137214d6-09d8-4a4f-bf37-fcef28d37d07/volumes" Oct 30 13:00:21.093394 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:21.093377 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3314d4c4-7905-42dd-b342-277da538ba35" path="/var/lib/kubelet/pods/3314d4c4-7905-42dd-b342-277da538ba35/volumes" Oct 30 13:00:21.093810 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:21.093795 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eb0a51c-3364-487b-8aec-77c2cb215dd7" path="/var/lib/kubelet/pods/9eb0a51c-3364-487b-8aec-77c2cb215dd7/volumes" Oct 30 13:00:21.094535 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:21.094516 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee548a01-d0b9-4387-a699-041ac6e9b205" path="/var/lib/kubelet/pods/ee548a01-d0b9-4387-a699-041ac6e9b205/volumes" Oct 30 13:00:21.575808 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:21.575771 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" Oct 30 13:00:21.576337 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:21.575771 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8" event={"ID":"de938051-680d-48df-80b7-8e71ee81217b","Type":"ContainerDied","Data":"57b57ae31b1dd93e0796fdf2945d59ce877a81f44cdaa6676fec5b1c8a527d2d"} Oct 30 13:00:21.576337 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:21.575988 4468 scope.go:122] "RemoveContainer" containerID="e9df172fda691cbaa94134661381ba4d50274d3c194a24a76e6fb7c9536fceff" Oct 30 13:00:21.643035 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:21.642993 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8"] Oct 30 13:00:21.650196 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:21.650161 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47fd99-29889420-fc9m8"] Oct 30 13:00:21.767909 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:21.767871 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4"] Oct 30 13:00:21.954472 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:21.954449 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" Oct 30 13:00:22.068412 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:22.068377 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778-maas-api-service-ca\") pod \"6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778\" (UID: \"6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778\") " Oct 30 13:00:22.068602 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:22.068562 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778-kube-api-access-9z7h8\" (UniqueName: \"kubernetes.io/projected/6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778-kube-api-access-9z7h8\") pod \"6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778\" (UID: \"6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778\") " Oct 30 13:00:22.068747 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:22.068723 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778-maas-api-service-ca" pod "6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778" (UID: "6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:22.069397 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:22.069371 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778-maas-api-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:00:22.070748 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:22.070723 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778-kube-api-access-9z7h8" pod "6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778" (UID: "6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778"). InnerVolumeSpecName "kube-api-access-9z7h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:22.171512 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:22.171473 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z7h8\" (UniqueName: \"kubernetes.io/projected/6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778-kube-api-access-9z7h8\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:00:22.592843 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:22.592767 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" Oct 30 13:00:22.592843 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:22.592806 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4" event={"ID":"6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778","Type":"ContainerDied","Data":"6de93e6f96804b252f3ba41ee569cf5accbb5ffa2a0a5ae79c69571da854670a"} Oct 30 13:00:22.593373 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:22.592872 4468 scope.go:122] "RemoveContainer" containerID="49a90f68f4e5724be7c72efea200ee481249d2f565a53efc31e6649c6bdc399f" Oct 30 13:00:22.657707 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:22.657678 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4"] Oct 30 13:00:22.669875 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:22.669847 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889420-mdkr4"] Oct 30 13:00:23.120566 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:23.120514 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778" path="/var/lib/kubelet/pods/6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778/volumes" Oct 30 13:00:23.121373 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:00:23.121348 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de938051-680d-48df-80b7-8e71ee81217b" path="/var/lib/kubelet/pods/de938051-680d-48df-80b7-8e71ee81217b/volumes" Oct 30 13:00:23.603608 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:23.603574 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:00:53.600429 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:00:53.600340 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:01:23.601102 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:01:23.601048 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:01:53.603212 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:01:53.603170 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:01:54.321222 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:01:54.321187 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj"] Oct 30 13:01:55.271268 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:01:55.271232 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb"] Oct 30 13:01:59.322215 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:01:59.322173 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" podUID="9002883e-d5ab-4331-b4fa-f3ae000c9784" containerName="payload-processing" containerID="cri-o://a23ecf1e7f95ac6cec818d6d562e2411e64fabb26ac74f2100093953b1ecf265" gracePeriod=25 Oct 30 13:01:59.605175 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:01:59.605145 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" Oct 30 13:01:59.658869 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:01:59.658829 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9002883e-d5ab-4331-b4fa-f3ae000c9784-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9002883e-d5ab-4331-b4fa-f3ae000c9784-routing-overlay-volume\") pod \"9002883e-d5ab-4331-b4fa-f3ae000c9784\" (UID: \"9002883e-d5ab-4331-b4fa-f3ae000c9784\") " Oct 30 13:01:59.659099 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:01:59.659047 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9002883e-d5ab-4331-b4fa-f3ae000c9784-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9002883e-d5ab-4331-b4fa-f3ae000c9784-provider-credentials\") pod \"9002883e-d5ab-4331-b4fa-f3ae000c9784\" (UID: \"9002883e-d5ab-4331-b4fa-f3ae000c9784\") " Oct 30 13:01:59.659223 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:01:59.659172 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9002883e-d5ab-4331-b4fa-f3ae000c9784-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9002883e-d5ab-4331-b4fa-f3ae000c9784-plugins-config-volume\") pod \"9002883e-d5ab-4331-b4fa-f3ae000c9784\" (UID: \"9002883e-d5ab-4331-b4fa-f3ae000c9784\") " Oct 30 13:01:59.659301 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:01:59.659283 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9002883e-d5ab-4331-b4fa-f3ae000c9784-kube-api-access-5vgbd\" (UniqueName: \"kubernetes.io/projected/9002883e-d5ab-4331-b4fa-f3ae000c9784-kube-api-access-5vgbd\") pod \"9002883e-d5ab-4331-b4fa-f3ae000c9784\" (UID: \"9002883e-d5ab-4331-b4fa-f3ae000c9784\") " Oct 30 13:01:59.660776 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:01:59.660745 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9002883e-d5ab-4331-b4fa-f3ae000c9784-routing-overlay-volume" pod "9002883e-d5ab-4331-b4fa-f3ae000c9784" (UID: "9002883e-d5ab-4331-b4fa-f3ae000c9784"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:01:59.661240 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:01:59.661212 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9002883e-d5ab-4331-b4fa-f3ae000c9784-plugins-config-volume" pod "9002883e-d5ab-4331-b4fa-f3ae000c9784" (UID: "9002883e-d5ab-4331-b4fa-f3ae000c9784"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:01:59.665592 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:01:59.665545 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9002883e-d5ab-4331-b4fa-f3ae000c9784-provider-credentials" pod "9002883e-d5ab-4331-b4fa-f3ae000c9784" (UID: "9002883e-d5ab-4331-b4fa-f3ae000c9784"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:01:59.677280 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:01:59.677245 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9002883e-d5ab-4331-b4fa-f3ae000c9784-kube-api-access-5vgbd" pod "9002883e-d5ab-4331-b4fa-f3ae000c9784" (UID: "9002883e-d5ab-4331-b4fa-f3ae000c9784"). InnerVolumeSpecName "kube-api-access-5vgbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:01:59.761826 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:01:59.761793 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9002883e-d5ab-4331-b4fa-f3ae000c9784-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:01:59.761826 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:01:59.761825 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9002883e-d5ab-4331-b4fa-f3ae000c9784-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:01:59.761826 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:01:59.761834 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9002883e-d5ab-4331-b4fa-f3ae000c9784-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:01:59.762055 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:01:59.761843 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vgbd\" (UniqueName: \"kubernetes.io/projected/9002883e-d5ab-4331-b4fa-f3ae000c9784-kube-api-access-5vgbd\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:00.272345 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.272300 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" podUID="0f9ddb0a-51f8-4d38-9605-74dc25689e46" containerName="payload-processing" containerID="cri-o://1f2a869c9ebdb3080cf8aa02005d8feaae5eb879461a6a37c6e741c49026c1c9" gracePeriod=25 Oct 30 13:02:00.404633 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.404605 4468 generic.go:356] "Generic (PLEG): container finished" podID="9002883e-d5ab-4331-b4fa-f3ae000c9784" containerID="a23ecf1e7f95ac6cec818d6d562e2411e64fabb26ac74f2100093953b1ecf265" exitCode=0 Oct 30 13:02:00.404993 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.404678 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" Oct 30 13:02:00.404993 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.404681 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" event={"ID":"9002883e-d5ab-4331-b4fa-f3ae000c9784","Type":"ContainerDied","Data":"a23ecf1e7f95ac6cec818d6d562e2411e64fabb26ac74f2100093953b1ecf265"} Oct 30 13:02:00.404993 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.404741 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj" event={"ID":"9002883e-d5ab-4331-b4fa-f3ae000c9784","Type":"ContainerDied","Data":"18e48fbc11a81a937f1e6f192007105d1576a9bb561f261291a3196caa6ee4e3"} Oct 30 13:02:00.404993 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.404765 4468 scope.go:122] "RemoveContainer" containerID="a23ecf1e7f95ac6cec818d6d562e2411e64fabb26ac74f2100093953b1ecf265" Oct 30 13:02:00.407612 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.407584 4468 generic.go:356] "Generic (PLEG): container finished" podID="0f9ddb0a-51f8-4d38-9605-74dc25689e46" containerID="1f2a869c9ebdb3080cf8aa02005d8feaae5eb879461a6a37c6e741c49026c1c9" exitCode=0 Oct 30 13:02:00.407748 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.407729 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" event={"ID":"0f9ddb0a-51f8-4d38-9605-74dc25689e46","Type":"ContainerDied","Data":"1f2a869c9ebdb3080cf8aa02005d8feaae5eb879461a6a37c6e741c49026c1c9"} Oct 30 13:02:00.418510 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.418489 4468 scope.go:122] "RemoveContainer" containerID="a23ecf1e7f95ac6cec818d6d562e2411e64fabb26ac74f2100093953b1ecf265" Oct 30 13:02:00.418842 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:02:00.418818 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a23ecf1e7f95ac6cec818d6d562e2411e64fabb26ac74f2100093953b1ecf265\": container with ID starting with a23ecf1e7f95ac6cec818d6d562e2411e64fabb26ac74f2100093953b1ecf265 not found: ID does not exist" containerID="a23ecf1e7f95ac6cec818d6d562e2411e64fabb26ac74f2100093953b1ecf265" Oct 30 13:02:00.418934 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.418852 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23ecf1e7f95ac6cec818d6d562e2411e64fabb26ac74f2100093953b1ecf265"} err="failed to get container status \"a23ecf1e7f95ac6cec818d6d562e2411e64fabb26ac74f2100093953b1ecf265\": rpc error: code = NotFound desc = could not find container \"a23ecf1e7f95ac6cec818d6d562e2411e64fabb26ac74f2100093953b1ecf265\": container with ID starting with a23ecf1e7f95ac6cec818d6d562e2411e64fabb26ac74f2100093953b1ecf265 not found: ID does not exist" Oct 30 13:02:00.465845 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.465814 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj"] Oct 30 13:02:00.477213 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.477184 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f7464f/payload-processing-external-model-e2e-extmod-a-f7464f-5f4fxh4vj"] Oct 30 13:02:00.597769 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.597745 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" Oct 30 13:02:00.675514 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.675477 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9ddb0a-51f8-4d38-9605-74dc25689e46-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f9ddb0a-51f8-4d38-9605-74dc25689e46-provider-credentials\") pod \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\" (UID: \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\") " Oct 30 13:02:00.675727 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.675705 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9ddb0a-51f8-4d38-9605-74dc25689e46-kube-api-access-v7l5m\" (UniqueName: \"kubernetes.io/projected/0f9ddb0a-51f8-4d38-9605-74dc25689e46-kube-api-access-v7l5m\") pod \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\" (UID: \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\") " Oct 30 13:02:00.675962 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.675940 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9ddb0a-51f8-4d38-9605-74dc25689e46-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f9ddb0a-51f8-4d38-9605-74dc25689e46-routing-overlay-volume\") pod \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\" (UID: \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\") " Oct 30 13:02:00.676483 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.676180 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9ddb0a-51f8-4d38-9605-74dc25689e46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9ddb0a-51f8-4d38-9605-74dc25689e46-plugins-config-volume\") pod \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\" (UID: \"0f9ddb0a-51f8-4d38-9605-74dc25689e46\") " Oct 30 13:02:00.676483 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.676405 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9ddb0a-51f8-4d38-9605-74dc25689e46-routing-overlay-volume" pod "0f9ddb0a-51f8-4d38-9605-74dc25689e46" (UID: "0f9ddb0a-51f8-4d38-9605-74dc25689e46"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:00.676671 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.676638 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9ddb0a-51f8-4d38-9605-74dc25689e46-plugins-config-volume" pod "0f9ddb0a-51f8-4d38-9605-74dc25689e46" (UID: "0f9ddb0a-51f8-4d38-9605-74dc25689e46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:00.677484 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.677431 4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f9ddb0a-51f8-4d38-9605-74dc25689e46-routing-overlay-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:00.677484 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.677461 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9ddb0a-51f8-4d38-9605-74dc25689e46-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:00.678678 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.678644 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9ddb0a-51f8-4d38-9605-74dc25689e46-provider-credentials" pod "0f9ddb0a-51f8-4d38-9605-74dc25689e46" (UID: "0f9ddb0a-51f8-4d38-9605-74dc25689e46"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:00.678784 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.678720 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9ddb0a-51f8-4d38-9605-74dc25689e46-kube-api-access-v7l5m" pod "0f9ddb0a-51f8-4d38-9605-74dc25689e46" (UID: "0f9ddb0a-51f8-4d38-9605-74dc25689e46"). InnerVolumeSpecName "kube-api-access-v7l5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:00.779181 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.779147 4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f9ddb0a-51f8-4d38-9605-74dc25689e46-provider-credentials\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:00.779181 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:00.779174 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7l5m\" (UniqueName: \"kubernetes.io/projected/0f9ddb0a-51f8-4d38-9605-74dc25689e46-kube-api-access-v7l5m\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:01.098622 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:01.098592 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9002883e-d5ab-4331-b4fa-f3ae000c9784" path="/var/lib/kubelet/pods/9002883e-d5ab-4331-b4fa-f3ae000c9784/volumes" Oct 30 13:02:01.416109 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:01.416077 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" Oct 30 13:02:01.416591 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:01.416111 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb" event={"ID":"0f9ddb0a-51f8-4d38-9605-74dc25689e46","Type":"ContainerDied","Data":"5250b6a245885801663fd4470d6ee009c529d34890ee1287a4f3577755e7f816"} Oct 30 13:02:01.416591 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:01.416184 4468 scope.go:122] "RemoveContainer" containerID="1f2a869c9ebdb3080cf8aa02005d8feaae5eb879461a6a37c6e741c49026c1c9" Oct 30 13:02:01.479048 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:01.479014 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb"] Oct 30 13:02:01.484644 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:01.484621 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-75cfl8czb"] Oct 30 13:02:03.094645 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:03.094609 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9ddb0a-51f8-4d38-9605-74dc25689e46" path="/var/lib/kubelet/pods/0f9ddb0a-51f8-4d38-9605-74dc25689e46/volumes" Oct 30 13:02:03.452033 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:03.452000 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk"] Oct 30 13:02:03.452304 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:03.452279 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" podUID="64cf635f-868a-4021-91aa-f36a314e26c1" containerName="istio-proxy" containerID="cri-o://fdca0830c58d7f99db6c6c6b0c2ffe1aab4f3312d7d6d0804897d1d59a434a65" gracePeriod=30 Oct 30 13:02:03.917203 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:03.917163 4468 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:02:03.917409 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:03.917245 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" podUID="64cf635f-868a-4021-91aa-f36a314e26c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:02:08.737049 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.736965 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 13:02:08.818401 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.818363 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64cf635f-868a-4021-91aa-f36a314e26c1-istio-token\" (UniqueName: \"kubernetes.io/projected/64cf635f-868a-4021-91aa-f36a314e26c1-istio-token\") pod \"64cf635f-868a-4021-91aa-f36a314e26c1\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " Oct 30 13:02:08.818593 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.818481 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64cf635f-868a-4021-91aa-f36a314e26c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64cf635f-868a-4021-91aa-f36a314e26c1-istio-podinfo\") pod \"64cf635f-868a-4021-91aa-f36a314e26c1\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " Oct 30 13:02:08.818593 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.818541 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64cf635f-868a-4021-91aa-f36a314e26c1-kube-api-access-vfzgc\" (UniqueName: \"kubernetes.io/projected/64cf635f-868a-4021-91aa-f36a314e26c1-kube-api-access-vfzgc\") pod \"64cf635f-868a-4021-91aa-f36a314e26c1\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " Oct 30 13:02:08.818719 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.818675 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-istio-data\") pod \"64cf635f-868a-4021-91aa-f36a314e26c1\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " Oct 30 13:02:08.818719 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.818713 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-workload-certs\") pod \"64cf635f-868a-4021-91aa-f36a314e26c1\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " Oct 30 13:02:08.818870 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.818852 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64cf635f-868a-4021-91aa-f36a314e26c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64cf635f-868a-4021-91aa-f36a314e26c1-istiod-ca-cert\") pod \"64cf635f-868a-4021-91aa-f36a314e26c1\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " Oct 30 13:02:08.818961 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.818932 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-credential-socket\") pod \"64cf635f-868a-4021-91aa-f36a314e26c1\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " Oct 30 13:02:08.818961 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.818939 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-istio-data" pod "64cf635f-868a-4021-91aa-f36a314e26c1" (UID: "64cf635f-868a-4021-91aa-f36a314e26c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:08.819094 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.818975 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-istio-envoy\") pod \"64cf635f-868a-4021-91aa-f36a314e26c1\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " Oct 30 13:02:08.819094 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.819035 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-workload-socket\") pod \"64cf635f-868a-4021-91aa-f36a314e26c1\" (UID: \"64cf635f-868a-4021-91aa-f36a314e26c1\") " Oct 30 13:02:08.819094 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.819038 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-workload-certs" pod "64cf635f-868a-4021-91aa-f36a314e26c1" (UID: "64cf635f-868a-4021-91aa-f36a314e26c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:08.819278 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.819214 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-credential-socket" pod "64cf635f-868a-4021-91aa-f36a314e26c1" (UID: "64cf635f-868a-4021-91aa-f36a314e26c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:08.819356 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.819317 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64cf635f-868a-4021-91aa-f36a314e26c1-istiod-ca-cert" pod "64cf635f-868a-4021-91aa-f36a314e26c1" (UID: "64cf635f-868a-4021-91aa-f36a314e26c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:08.819513 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.819489 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-workload-socket" pod "64cf635f-868a-4021-91aa-f36a314e26c1" (UID: "64cf635f-868a-4021-91aa-f36a314e26c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:08.819920 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.819892 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64cf635f-868a-4021-91aa-f36a314e26c1-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:08.820032 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.819923 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:08.820032 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.819942 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:08.820032 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.819955 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:08.820032 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.819969 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:08.821025 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.820991 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64cf635f-868a-4021-91aa-f36a314e26c1-kube-api-access-vfzgc" pod "64cf635f-868a-4021-91aa-f36a314e26c1" (UID: "64cf635f-868a-4021-91aa-f36a314e26c1"). InnerVolumeSpecName "kube-api-access-vfzgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:08.821156 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.821086 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64cf635f-868a-4021-91aa-f36a314e26c1-istio-token" pod "64cf635f-868a-4021-91aa-f36a314e26c1" (UID: "64cf635f-868a-4021-91aa-f36a314e26c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:08.821307 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.821291 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64cf635f-868a-4021-91aa-f36a314e26c1-istio-podinfo" pod "64cf635f-868a-4021-91aa-f36a314e26c1" (UID: "64cf635f-868a-4021-91aa-f36a314e26c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:02:08.821526 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.821499 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-istio-envoy" pod "64cf635f-868a-4021-91aa-f36a314e26c1" (UID: "64cf635f-868a-4021-91aa-f36a314e26c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:08.921444 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.921410 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64cf635f-868a-4021-91aa-f36a314e26c1-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:08.921444 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.921442 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64cf635f-868a-4021-91aa-f36a314e26c1-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:08.921444 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.921452 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfzgc\" (UniqueName: \"kubernetes.io/projected/64cf635f-868a-4021-91aa-f36a314e26c1-kube-api-access-vfzgc\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:08.921693 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:08.921462 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64cf635f-868a-4021-91aa-f36a314e26c1-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:09.488187 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:09.488150 4468 generic.go:356] "Generic (PLEG): container finished" podID="64cf635f-868a-4021-91aa-f36a314e26c1" containerID="fdca0830c58d7f99db6c6c6b0c2ffe1aab4f3312d7d6d0804897d1d59a434a65" exitCode=0 Oct 30 13:02:09.488381 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:09.488196 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" Oct 30 13:02:09.488381 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:09.488196 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" event={"ID":"64cf635f-868a-4021-91aa-f36a314e26c1","Type":"ContainerDied","Data":"fdca0830c58d7f99db6c6c6b0c2ffe1aab4f3312d7d6d0804897d1d59a434a65"} Oct 30 13:02:09.488381 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:09.488325 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk" event={"ID":"64cf635f-868a-4021-91aa-f36a314e26c1","Type":"ContainerDied","Data":"0054c83a779898c0b1bf125e8aeca0daa3f2bc2eecc6c9527a4e045ab5afe915"} Oct 30 13:02:09.488381 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:09.488353 4468 scope.go:122] "RemoveContainer" containerID="fdca0830c58d7f99db6c6c6b0c2ffe1aab4f3312d7d6d0804897d1d59a434a65" Oct 30 13:02:09.499434 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:09.499415 4468 scope.go:122] "RemoveContainer" containerID="fdca0830c58d7f99db6c6c6b0c2ffe1aab4f3312d7d6d0804897d1d59a434a65" Oct 30 13:02:09.499709 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:02:09.499690 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdca0830c58d7f99db6c6c6b0c2ffe1aab4f3312d7d6d0804897d1d59a434a65\": container with ID starting with fdca0830c58d7f99db6c6c6b0c2ffe1aab4f3312d7d6d0804897d1d59a434a65 not found: ID does not exist" containerID="fdca0830c58d7f99db6c6c6b0c2ffe1aab4f3312d7d6d0804897d1d59a434a65" Oct 30 13:02:09.499765 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:09.499717 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdca0830c58d7f99db6c6c6b0c2ffe1aab4f3312d7d6d0804897d1d59a434a65"} err="failed to get container status \"fdca0830c58d7f99db6c6c6b0c2ffe1aab4f3312d7d6d0804897d1d59a434a65\": rpc error: code = NotFound desc = could not find container \"fdca0830c58d7f99db6c6c6b0c2ffe1aab4f3312d7d6d0804897d1d59a434a65\": container with ID starting with fdca0830c58d7f99db6c6c6b0c2ffe1aab4f3312d7d6d0804897d1d59a434a65 not found: ID does not exist" Oct 30 13:02:09.561028 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:09.560994 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk"] Oct 30 13:02:09.574055 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:09.574029 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vjsqk"] Oct 30 13:02:11.093171 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:11.093141 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64cf635f-868a-4021-91aa-f36a314e26c1" path="/var/lib/kubelet/pods/64cf635f-868a-4021-91aa-f36a314e26c1/volumes" Oct 30 13:02:12.505813 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:12.505769 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r"] Oct 30 13:02:12.541231 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:12.541201 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9"] Oct 30 13:02:17.506235 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:17.506178 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" podUID="7f2bdcd0-98ce-4c5b-bb1b-81db35f39186" containerName="payload-processing" containerID="cri-o://e9e6b3e52ecc3dde7976c2aae31130b82f0188c25a2edff33b924efe68ae99e1" gracePeriod=25 Oct 30 13:02:17.541624 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:17.541585 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" podUID="5ae0dc66-24bc-4aad-a275-bc7a8a2c3206" containerName="payload-pre-processing" containerID="cri-o://b430a7f1d24b7e8e908d15f04298d8f415b2cda5bdc4a08e350a9fc9189ea17a" gracePeriod=25 Oct 30 13:02:23.600682 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:02:23.600594 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:02:26.760161 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:26.760115 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" podUID="7f2bdcd0-98ce-4c5b-bb1b-81db35f39186" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:02:26.760734 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:26.760128 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" podUID="5ae0dc66-24bc-4aad-a275-bc7a8a2c3206" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:02:34.402060 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:34.402022 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj"] Oct 30 13:02:34.402579 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:34.402223 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" podUID="16dae47c-220c-4d83-89e5-52d1746775e8" containerName="istio-proxy" containerID="cri-o://7cbeeb3b2b35a7e456f12b1594e679754a5929d5a07785c074026e1e873caf71" gracePeriod=30 Oct 30 13:02:35.395581 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:35.395540 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl"] Oct 30 13:02:35.395809 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:35.395784 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" podUID="1270203a-f11c-4b92-9cb6-b76ab1bbc2a4" containerName="maas-api" containerID="cri-o://30026e6efbe060567e62c6138a412e1c204b95ef760d9d376e33914efba76a14" gracePeriod=30 Oct 30 13:02:35.703150 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:35.702847 4468 generic.go:356] "Generic (PLEG): container finished" podID="1270203a-f11c-4b92-9cb6-b76ab1bbc2a4" containerID="30026e6efbe060567e62c6138a412e1c204b95ef760d9d376e33914efba76a14" exitCode=0 Oct 30 13:02:35.703150 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:35.703025 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" event={"ID":"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4","Type":"ContainerDied","Data":"30026e6efbe060567e62c6138a412e1c204b95ef760d9d376e33914efba76a14"} Oct 30 13:02:35.745799 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:35.745766 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" Oct 30 13:02:35.881847 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:35.881803 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-kube-api-access-ljpn7\" (UniqueName: \"kubernetes.io/projected/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-kube-api-access-ljpn7\") pod \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\" (UID: \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\") " Oct 30 13:02:35.882151 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:35.882126 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-maas-api-tls\") pod \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\" (UID: \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\") " Oct 30 13:02:35.882393 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:35.882374 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-metrics-certs\") pod \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\" (UID: \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\") " Oct 30 13:02:35.882589 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:35.882572 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-openshift-service-ca\") pod \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\" (UID: \"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4\") " Oct 30 13:02:35.891110 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:35.884099 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-openshift-service-ca" pod "1270203a-f11c-4b92-9cb6-b76ab1bbc2a4" (UID: "1270203a-f11c-4b92-9cb6-b76ab1bbc2a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:35.899645 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:35.899604 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-maas-api-tls" pod "1270203a-f11c-4b92-9cb6-b76ab1bbc2a4" (UID: "1270203a-f11c-4b92-9cb6-b76ab1bbc2a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:02:35.902407 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:35.902341 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-kube-api-access-ljpn7" pod "1270203a-f11c-4b92-9cb6-b76ab1bbc2a4" (UID: "1270203a-f11c-4b92-9cb6-b76ab1bbc2a4"). InnerVolumeSpecName "kube-api-access-ljpn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:35.907017 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:35.906965 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-metrics-certs" pod "1270203a-f11c-4b92-9cb6-b76ab1bbc2a4" (UID: "1270203a-f11c-4b92-9cb6-b76ab1bbc2a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:02:35.988139 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:35.984995 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljpn7\" (UniqueName: \"kubernetes.io/projected/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-kube-api-access-ljpn7\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:35.988139 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:35.985032 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:35.988139 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:35.985047 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:35.988139 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:35.985060 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:36.712508 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:36.712478 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" Oct 30 13:02:36.712941 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:36.712504 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl" event={"ID":"1270203a-f11c-4b92-9cb6-b76ab1bbc2a4","Type":"ContainerDied","Data":"5ac6bd22277d62da4e67c6ae9bfb1d6b840b9cbdf6bfe7e740f386cd4cc4cd1e"} Oct 30 13:02:36.712941 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:36.712568 4468 scope.go:122] "RemoveContainer" containerID="30026e6efbe060567e62c6138a412e1c204b95ef760d9d376e33914efba76a14" Oct 30 13:02:36.759959 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:36.759917 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" podUID="5ae0dc66-24bc-4aad-a275-bc7a8a2c3206" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:02:36.760140 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:36.759917 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" podUID="7f2bdcd0-98ce-4c5b-bb1b-81db35f39186" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:02:36.775527 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:36.775496 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl"] Oct 30 13:02:36.780827 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:36.780801 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f7464f-7c9f76bb88-59sgl"] Oct 30 13:02:37.092659 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:37.092580 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1270203a-f11c-4b92-9cb6-b76ab1bbc2a4" path="/var/lib/kubelet/pods/1270203a-f11c-4b92-9cb6-b76ab1bbc2a4/volumes" Oct 30 13:02:37.735568 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:37.734657 4468 generic.go:356] "Generic (PLEG): container finished" podID="5ae0dc66-24bc-4aad-a275-bc7a8a2c3206" containerID="b430a7f1d24b7e8e908d15f04298d8f415b2cda5bdc4a08e350a9fc9189ea17a" exitCode=0 Oct 30 13:02:37.735568 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:37.734802 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" event={"ID":"5ae0dc66-24bc-4aad-a275-bc7a8a2c3206","Type":"ContainerDied","Data":"b430a7f1d24b7e8e908d15f04298d8f415b2cda5bdc4a08e350a9fc9189ea17a"} Oct 30 13:02:37.738425 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:37.738271 4468 generic.go:356] "Generic (PLEG): container finished" podID="7f2bdcd0-98ce-4c5b-bb1b-81db35f39186" containerID="e9e6b3e52ecc3dde7976c2aae31130b82f0188c25a2edff33b924efe68ae99e1" exitCode=0 Oct 30 13:02:37.738425 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:37.738363 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" event={"ID":"7f2bdcd0-98ce-4c5b-bb1b-81db35f39186","Type":"ContainerDied","Data":"e9e6b3e52ecc3dde7976c2aae31130b82f0188c25a2edff33b924efe68ae99e1"} Oct 30 13:02:37.845697 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:37.845669 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" Oct 30 13:02:37.872762 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:37.872740 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" Oct 30 13:02:37.922316 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:37.922275 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f2bdcd0-98ce-4c5b-bb1b-81db35f39186-kube-api-access-tpfbt\" (UniqueName: \"kubernetes.io/projected/7f2bdcd0-98ce-4c5b-bb1b-81db35f39186-kube-api-access-tpfbt\") pod \"7f2bdcd0-98ce-4c5b-bb1b-81db35f39186\" (UID: \"7f2bdcd0-98ce-4c5b-bb1b-81db35f39186\") " Oct 30 13:02:37.922507 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:37.922421 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f2bdcd0-98ce-4c5b-bb1b-81db35f39186-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f2bdcd0-98ce-4c5b-bb1b-81db35f39186-plugins-config-volume\") pod \"7f2bdcd0-98ce-4c5b-bb1b-81db35f39186\" (UID: \"7f2bdcd0-98ce-4c5b-bb1b-81db35f39186\") " Oct 30 13:02:37.922897 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:37.922867 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2bdcd0-98ce-4c5b-bb1b-81db35f39186-plugins-config-volume" pod "7f2bdcd0-98ce-4c5b-bb1b-81db35f39186" (UID: "7f2bdcd0-98ce-4c5b-bb1b-81db35f39186"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:37.923349 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:37.923322 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f2bdcd0-98ce-4c5b-bb1b-81db35f39186-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:37.924609 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:37.924585 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2bdcd0-98ce-4c5b-bb1b-81db35f39186-kube-api-access-tpfbt" pod "7f2bdcd0-98ce-4c5b-bb1b-81db35f39186" (UID: "7f2bdcd0-98ce-4c5b-bb1b-81db35f39186"). InnerVolumeSpecName "kube-api-access-tpfbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:38.023947 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:38.023915 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae0dc66-24bc-4aad-a275-bc7a8a2c3206-kube-api-access-tkbng\" (UniqueName: \"kubernetes.io/projected/5ae0dc66-24bc-4aad-a275-bc7a8a2c3206-kube-api-access-tkbng\") pod \"5ae0dc66-24bc-4aad-a275-bc7a8a2c3206\" (UID: \"5ae0dc66-24bc-4aad-a275-bc7a8a2c3206\") " Oct 30 13:02:38.024171 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:38.023957 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae0dc66-24bc-4aad-a275-bc7a8a2c3206-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae0dc66-24bc-4aad-a275-bc7a8a2c3206-plugins-config-volume\") pod \"5ae0dc66-24bc-4aad-a275-bc7a8a2c3206\" (UID: \"5ae0dc66-24bc-4aad-a275-bc7a8a2c3206\") " Oct 30 13:02:38.024387 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:38.024362 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae0dc66-24bc-4aad-a275-bc7a8a2c3206-plugins-config-volume" pod "5ae0dc66-24bc-4aad-a275-bc7a8a2c3206" (UID: "5ae0dc66-24bc-4aad-a275-bc7a8a2c3206"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:38.024655 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:38.024635 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpfbt\" (UniqueName: \"kubernetes.io/projected/7f2bdcd0-98ce-4c5b-bb1b-81db35f39186-kube-api-access-tpfbt\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:38.024728 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:38.024658 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae0dc66-24bc-4aad-a275-bc7a8a2c3206-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:38.026149 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:38.026128 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae0dc66-24bc-4aad-a275-bc7a8a2c3206-kube-api-access-tkbng" pod "5ae0dc66-24bc-4aad-a275-bc7a8a2c3206" (UID: "5ae0dc66-24bc-4aad-a275-bc7a8a2c3206"). InnerVolumeSpecName "kube-api-access-tkbng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:38.125815 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:38.125735 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkbng\" (UniqueName: \"kubernetes.io/projected/5ae0dc66-24bc-4aad-a275-bc7a8a2c3206-kube-api-access-tkbng\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:38.749258 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:38.749226 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" Oct 30 13:02:38.749700 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:38.749269 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9" event={"ID":"5ae0dc66-24bc-4aad-a275-bc7a8a2c3206","Type":"ContainerDied","Data":"a0653dd9e5b8c39838529534778d9b2ce8ec1d7568a3213b555359979c5e2589"} Oct 30 13:02:38.749700 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:38.749333 4468 scope.go:122] "RemoveContainer" containerID="b430a7f1d24b7e8e908d15f04298d8f415b2cda5bdc4a08e350a9fc9189ea17a" Oct 30 13:02:38.750891 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:38.750868 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" Oct 30 13:02:38.750891 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:38.750872 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r" event={"ID":"7f2bdcd0-98ce-4c5b-bb1b-81db35f39186","Type":"ContainerDied","Data":"b64347a4e24f4eee4ac9640e6732b3f1f4545bde1439502879eb9a3a70dfd4ae"} Oct 30 13:02:38.761971 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:38.761953 4468 scope.go:122] "RemoveContainer" containerID="e9e6b3e52ecc3dde7976c2aae31130b82f0188c25a2edff33b924efe68ae99e1" Oct 30 13:02:38.809459 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:38.809387 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r"] Oct 30 13:02:38.823082 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:38.823036 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f7464f-7bb7854df4-5mz6r"] Oct 30 13:02:38.861279 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:38.861242 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9"] Oct 30 13:02:38.866152 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:38.866127 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f7464f-799f6f7cdc-j5lv9"] Oct 30 13:02:39.091722 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.091637 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae0dc66-24bc-4aad-a275-bc7a8a2c3206" path="/var/lib/kubelet/pods/5ae0dc66-24bc-4aad-a275-bc7a8a2c3206/volumes" Oct 30 13:02:39.092137 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.092122 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f2bdcd0-98ce-4c5b-bb1b-81db35f39186" path="/var/lib/kubelet/pods/7f2bdcd0-98ce-4c5b-bb1b-81db35f39186/volumes" Oct 30 13:02:39.720199 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.720171 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 13:02:39.761080 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.761036 4468 generic.go:356] "Generic (PLEG): container finished" podID="16dae47c-220c-4d83-89e5-52d1746775e8" containerID="7cbeeb3b2b35a7e456f12b1594e679754a5929d5a07785c074026e1e873caf71" exitCode=0 Oct 30 13:02:39.761535 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.761108 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" Oct 30 13:02:39.761535 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.761116 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" event={"ID":"16dae47c-220c-4d83-89e5-52d1746775e8","Type":"ContainerDied","Data":"7cbeeb3b2b35a7e456f12b1594e679754a5929d5a07785c074026e1e873caf71"} Oct 30 13:02:39.761535 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.761181 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj" event={"ID":"16dae47c-220c-4d83-89e5-52d1746775e8","Type":"ContainerDied","Data":"903900eb4b6cbc870e30a3d9e94a45590f3cb2beed6c7f6463318f362144064d"} Oct 30 13:02:39.761535 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.761213 4468 scope.go:122] "RemoveContainer" containerID="7cbeeb3b2b35a7e456f12b1594e679754a5929d5a07785c074026e1e873caf71" Oct 30 13:02:39.776456 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.776438 4468 scope.go:122] "RemoveContainer" containerID="7cbeeb3b2b35a7e456f12b1594e679754a5929d5a07785c074026e1e873caf71" Oct 30 13:02:39.776741 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:02:39.776721 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cbeeb3b2b35a7e456f12b1594e679754a5929d5a07785c074026e1e873caf71\": container with ID starting with 7cbeeb3b2b35a7e456f12b1594e679754a5929d5a07785c074026e1e873caf71 not found: ID does not exist" containerID="7cbeeb3b2b35a7e456f12b1594e679754a5929d5a07785c074026e1e873caf71" Oct 30 13:02:39.776795 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.776750 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cbeeb3b2b35a7e456f12b1594e679754a5929d5a07785c074026e1e873caf71"} err="failed to get container status \"7cbeeb3b2b35a7e456f12b1594e679754a5929d5a07785c074026e1e873caf71\": rpc error: code = NotFound desc = could not find container \"7cbeeb3b2b35a7e456f12b1594e679754a5929d5a07785c074026e1e873caf71\": container with ID starting with 7cbeeb3b2b35a7e456f12b1594e679754a5929d5a07785c074026e1e873caf71 not found: ID does not exist" Oct 30 13:02:39.851635 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.851595 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-credential-socket\") pod \"16dae47c-220c-4d83-89e5-52d1746775e8\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " Oct 30 13:02:39.851850 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.851647 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16dae47c-220c-4d83-89e5-52d1746775e8-kube-api-access-9rrtl\" (UniqueName: \"kubernetes.io/projected/16dae47c-220c-4d83-89e5-52d1746775e8-kube-api-access-9rrtl\") pod \"16dae47c-220c-4d83-89e5-52d1746775e8\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " Oct 30 13:02:39.851850 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.851681 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-istio-data\") pod \"16dae47c-220c-4d83-89e5-52d1746775e8\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " Oct 30 13:02:39.851850 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.851700 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-istio-envoy\") pod \"16dae47c-220c-4d83-89e5-52d1746775e8\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " Oct 30 13:02:39.851850 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.851742 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16dae47c-220c-4d83-89e5-52d1746775e8-istio-token\" (UniqueName: \"kubernetes.io/projected/16dae47c-220c-4d83-89e5-52d1746775e8-istio-token\") pod \"16dae47c-220c-4d83-89e5-52d1746775e8\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " Oct 30 13:02:39.851850 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.851810 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-workload-certs\") pod \"16dae47c-220c-4d83-89e5-52d1746775e8\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " Oct 30 13:02:39.852106 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.851920 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-credential-socket" pod "16dae47c-220c-4d83-89e5-52d1746775e8" (UID: "16dae47c-220c-4d83-89e5-52d1746775e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.852106 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.852096 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16dae47c-220c-4d83-89e5-52d1746775e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dae47c-220c-4d83-89e5-52d1746775e8-istio-podinfo\") pod \"16dae47c-220c-4d83-89e5-52d1746775e8\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " Oct 30 13:02:39.852242 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.852144 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-workload-socket\") pod \"16dae47c-220c-4d83-89e5-52d1746775e8\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " Oct 30 13:02:39.852242 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.852162 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16dae47c-220c-4d83-89e5-52d1746775e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dae47c-220c-4d83-89e5-52d1746775e8-istiod-ca-cert\") pod \"16dae47c-220c-4d83-89e5-52d1746775e8\" (UID: \"16dae47c-220c-4d83-89e5-52d1746775e8\") " Oct 30 13:02:39.852444 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.852414 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-workload-certs" pod "16dae47c-220c-4d83-89e5-52d1746775e8" (UID: "16dae47c-220c-4d83-89e5-52d1746775e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.852541 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.852519 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-workload-socket" pod "16dae47c-220c-4d83-89e5-52d1746775e8" (UID: "16dae47c-220c-4d83-89e5-52d1746775e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.852635 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.852599 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-istio-data" pod "16dae47c-220c-4d83-89e5-52d1746775e8" (UID: "16dae47c-220c-4d83-89e5-52d1746775e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.852782 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.852742 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16dae47c-220c-4d83-89e5-52d1746775e8-istiod-ca-cert" pod "16dae47c-220c-4d83-89e5-52d1746775e8" (UID: "16dae47c-220c-4d83-89e5-52d1746775e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:39.852962 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.852935 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:39.853140 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.852968 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dae47c-220c-4d83-89e5-52d1746775e8-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:39.853140 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.852985 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:39.853140 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.853001 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:39.853140 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.853016 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:39.854405 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.854379 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16dae47c-220c-4d83-89e5-52d1746775e8-istio-token" pod "16dae47c-220c-4d83-89e5-52d1746775e8" (UID: "16dae47c-220c-4d83-89e5-52d1746775e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:39.854568 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.854552 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-istio-envoy" pod "16dae47c-220c-4d83-89e5-52d1746775e8" (UID: "16dae47c-220c-4d83-89e5-52d1746775e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:39.854640 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.854570 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16dae47c-220c-4d83-89e5-52d1746775e8-kube-api-access-9rrtl" pod "16dae47c-220c-4d83-89e5-52d1746775e8" (UID: "16dae47c-220c-4d83-89e5-52d1746775e8"). InnerVolumeSpecName "kube-api-access-9rrtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:39.854640 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.854611 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16dae47c-220c-4d83-89e5-52d1746775e8-istio-podinfo" pod "16dae47c-220c-4d83-89e5-52d1746775e8" (UID: "16dae47c-220c-4d83-89e5-52d1746775e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:02:39.954746 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.954644 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dae47c-220c-4d83-89e5-52d1746775e8-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:39.954746 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.954688 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rrtl\" (UniqueName: \"kubernetes.io/projected/16dae47c-220c-4d83-89e5-52d1746775e8-kube-api-access-9rrtl\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:39.954746 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.954704 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dae47c-220c-4d83-89e5-52d1746775e8-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:39.954746 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:39.954718 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16dae47c-220c-4d83-89e5-52d1746775e8-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:40.116053 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:40.116014 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj"] Oct 30 13:02:40.129626 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:40.129598 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qtfcj"] Oct 30 13:02:41.092533 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:41.092498 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16dae47c-220c-4d83-89e5-52d1746775e8" path="/var/lib/kubelet/pods/16dae47c-220c-4d83-89e5-52d1746775e8/volumes" Oct 30 13:02:44.660827 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:44.660786 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts"] Oct 30 13:02:44.661343 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:44.661052 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" podUID="0b15b44d-dfe6-4001-966b-f72c56f449c1" containerName="istio-proxy" containerID="cri-o://1e446782f77f57beb977f27cf067f78272711ef1b394eff8084889ea914d650b" gracePeriod=30 Oct 30 13:02:46.742534 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:46.742496 4468 patch_prober.go:29] interesting pod/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:02:46.742930 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:46.742559 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" podUID="0b15b44d-dfe6-4001-966b-f72c56f449c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:02:49.861822 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:49.860202 4468 generic.go:356] "Generic (PLEG): container finished" podID="0b15b44d-dfe6-4001-966b-f72c56f449c1" containerID="1e446782f77f57beb977f27cf067f78272711ef1b394eff8084889ea914d650b" exitCode=0 Oct 30 13:02:49.861822 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:49.860388 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" event={"ID":"0b15b44d-dfe6-4001-966b-f72c56f449c1","Type":"ContainerDied","Data":"1e446782f77f57beb977f27cf067f78272711ef1b394eff8084889ea914d650b"} Oct 30 13:02:49.861822 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:49.860453 4468 scope.go:122] "RemoveContainer" containerID="c26876500b1851d4d59a53365d3ee95f0d5a608feb2148d066743c2686108c5c" Oct 30 13:02:49.953035 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:49.952969 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 13:02:50.011311 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.011271 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-workload-socket\") pod \"0b15b44d-dfe6-4001-966b-f72c56f449c1\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " Oct 30 13:02:50.011465 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.011442 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-podinfo\") pod \"0b15b44d-dfe6-4001-966b-f72c56f449c1\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " Oct 30 13:02:50.011509 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.011481 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-credential-socket\") pod \"0b15b44d-dfe6-4001-966b-f72c56f449c1\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " Oct 30 13:02:50.011561 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.011531 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b15b44d-dfe6-4001-966b-f72c56f449c1-kube-api-access-5bpx6\" (UniqueName: \"kubernetes.io/projected/0b15b44d-dfe6-4001-966b-f72c56f449c1-kube-api-access-5bpx6\") pod \"0b15b44d-dfe6-4001-966b-f72c56f449c1\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " Oct 30 13:02:50.011561 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.011531 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-workload-socket" pod "0b15b44d-dfe6-4001-966b-f72c56f449c1" (UID: "0b15b44d-dfe6-4001-966b-f72c56f449c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:50.011665 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.011565 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-envoy\") pod \"0b15b44d-dfe6-4001-966b-f72c56f449c1\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " Oct 30 13:02:50.011665 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.011605 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b15b44d-dfe6-4001-966b-f72c56f449c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b15b44d-dfe6-4001-966b-f72c56f449c1-istiod-ca-cert\") pod \"0b15b44d-dfe6-4001-966b-f72c56f449c1\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " Oct 30 13:02:50.011781 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.011666 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-workload-certs\") pod \"0b15b44d-dfe6-4001-966b-f72c56f449c1\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " Oct 30 13:02:50.011781 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.011724 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-token\" (UniqueName: \"kubernetes.io/projected/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-token\") pod \"0b15b44d-dfe6-4001-966b-f72c56f449c1\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " Oct 30 13:02:50.011781 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.011758 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-credential-socket" pod "0b15b44d-dfe6-4001-966b-f72c56f449c1" (UID: "0b15b44d-dfe6-4001-966b-f72c56f449c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:50.011781 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.011765 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-data\") pod \"0b15b44d-dfe6-4001-966b-f72c56f449c1\" (UID: \"0b15b44d-dfe6-4001-966b-f72c56f449c1\") " Oct 30 13:02:50.011997 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.011968 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-data" pod "0b15b44d-dfe6-4001-966b-f72c56f449c1" (UID: "0b15b44d-dfe6-4001-966b-f72c56f449c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:50.012087 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.012042 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-workload-certs" pod "0b15b44d-dfe6-4001-966b-f72c56f449c1" (UID: "0b15b44d-dfe6-4001-966b-f72c56f449c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:50.012399 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.012374 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b15b44d-dfe6-4001-966b-f72c56f449c1-istiod-ca-cert" pod "0b15b44d-dfe6-4001-966b-f72c56f449c1" (UID: "0b15b44d-dfe6-4001-966b-f72c56f449c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:50.012550 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.012533 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:50.012622 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.012552 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:50.012622 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.012561 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:50.012622 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.012569 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b15b44d-dfe6-4001-966b-f72c56f449c1-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:50.012622 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.012578 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:50.013985 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.013939 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-envoy" pod "0b15b44d-dfe6-4001-966b-f72c56f449c1" (UID: "0b15b44d-dfe6-4001-966b-f72c56f449c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:50.014148 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.014108 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b15b44d-dfe6-4001-966b-f72c56f449c1-kube-api-access-5bpx6" pod "0b15b44d-dfe6-4001-966b-f72c56f449c1" (UID: "0b15b44d-dfe6-4001-966b-f72c56f449c1"). InnerVolumeSpecName "kube-api-access-5bpx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:50.014396 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.014371 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-token" pod "0b15b44d-dfe6-4001-966b-f72c56f449c1" (UID: "0b15b44d-dfe6-4001-966b-f72c56f449c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:50.014632 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.014610 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-podinfo" pod "0b15b44d-dfe6-4001-966b-f72c56f449c1" (UID: "0b15b44d-dfe6-4001-966b-f72c56f449c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:02:50.114247 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.114213 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:50.114247 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.114239 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bpx6\" (UniqueName: \"kubernetes.io/projected/0b15b44d-dfe6-4001-966b-f72c56f449c1-kube-api-access-5bpx6\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:50.114247 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.114249 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:50.114505 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.114261 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b15b44d-dfe6-4001-966b-f72c56f449c1-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:02:50.870236 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.870204 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" event={"ID":"0b15b44d-dfe6-4001-966b-f72c56f449c1","Type":"ContainerDied","Data":"4870ec8c883e210108ddc1af4e14c2c5d278b8bac26833d3b07f4c1de41df14c"} Oct 30 13:02:50.870236 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.870232 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts" Oct 30 13:02:50.870751 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.870256 4468 scope.go:122] "RemoveContainer" containerID="1e446782f77f57beb977f27cf067f78272711ef1b394eff8084889ea914d650b" Oct 30 13:02:50.944032 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.944004 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts"] Oct 30 13:02:50.956395 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:50.956368 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-f7464f-openshift-default-696b559f45-jm5ts"] Oct 30 13:02:51.092140 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:51.092111 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b15b44d-dfe6-4001-966b-f72c56f449c1" path="/var/lib/kubelet/pods/0b15b44d-dfe6-4001-966b-f72c56f449c1/volumes" Oct 30 13:02:53.601089 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:02:53.601032 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:02:55.782396 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:02:55.782356 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq"] Oct 30 13:03:00.782889 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:00.782832 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" podUID="ba1e2c95-cd0f-43dc-89be-607ea62fd51d" containerName="payload-processing" containerID="cri-o://50bf3c9977b80822fb7d885e5be7ac2584302ced6f4b98ef1f7bce05dac6dfc0" gracePeriod=25 Oct 30 13:03:09.693451 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:09.693407 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" podUID="ba1e2c95-cd0f-43dc-89be-607ea62fd51d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:03:19.693064 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:19.693025 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" podUID="ba1e2c95-cd0f-43dc-89be-607ea62fd51d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:03:21.094401 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:21.094375 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" Oct 30 13:03:21.125939 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:21.125904 4468 generic.go:356] "Generic (PLEG): container finished" podID="ba1e2c95-cd0f-43dc-89be-607ea62fd51d" containerID="50bf3c9977b80822fb7d885e5be7ac2584302ced6f4b98ef1f7bce05dac6dfc0" exitCode=0 Oct 30 13:03:21.126106 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:21.125944 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" Oct 30 13:03:21.126106 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:21.125937 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" event={"ID":"ba1e2c95-cd0f-43dc-89be-607ea62fd51d","Type":"ContainerDied","Data":"50bf3c9977b80822fb7d885e5be7ac2584302ced6f4b98ef1f7bce05dac6dfc0"} Oct 30 13:03:21.126106 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:21.126088 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq" event={"ID":"ba1e2c95-cd0f-43dc-89be-607ea62fd51d","Type":"ContainerDied","Data":"8568aabad39c3b0ed7329f79588d246ea330f8ded646557138905157f50cf0b1"} Oct 30 13:03:21.126235 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:21.126119 4468 scope.go:122] "RemoveContainer" containerID="50bf3c9977b80822fb7d885e5be7ac2584302ced6f4b98ef1f7bce05dac6dfc0" Oct 30 13:03:21.147381 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:21.147352 4468 scope.go:122] "RemoveContainer" containerID="50bf3c9977b80822fb7d885e5be7ac2584302ced6f4b98ef1f7bce05dac6dfc0" Oct 30 13:03:21.149170 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:03:21.149101 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50bf3c9977b80822fb7d885e5be7ac2584302ced6f4b98ef1f7bce05dac6dfc0\": container with ID starting with 50bf3c9977b80822fb7d885e5be7ac2584302ced6f4b98ef1f7bce05dac6dfc0 not found: ID does not exist" containerID="50bf3c9977b80822fb7d885e5be7ac2584302ced6f4b98ef1f7bce05dac6dfc0" Oct 30 13:03:21.149170 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:21.149139 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50bf3c9977b80822fb7d885e5be7ac2584302ced6f4b98ef1f7bce05dac6dfc0"} err="failed to get container status \"50bf3c9977b80822fb7d885e5be7ac2584302ced6f4b98ef1f7bce05dac6dfc0\": rpc error: code = NotFound desc = could not find container \"50bf3c9977b80822fb7d885e5be7ac2584302ced6f4b98ef1f7bce05dac6dfc0\": container with ID starting with 50bf3c9977b80822fb7d885e5be7ac2584302ced6f4b98ef1f7bce05dac6dfc0 not found: ID does not exist" Oct 30 13:03:21.252278 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:21.252242 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba1e2c95-cd0f-43dc-89be-607ea62fd51d-kube-api-access-bqpr6\" (UniqueName: \"kubernetes.io/projected/ba1e2c95-cd0f-43dc-89be-607ea62fd51d-kube-api-access-bqpr6\") pod \"ba1e2c95-cd0f-43dc-89be-607ea62fd51d\" (UID: \"ba1e2c95-cd0f-43dc-89be-607ea62fd51d\") " Oct 30 13:03:21.252466 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:21.252413 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba1e2c95-cd0f-43dc-89be-607ea62fd51d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba1e2c95-cd0f-43dc-89be-607ea62fd51d-plugins-config-volume\") pod \"ba1e2c95-cd0f-43dc-89be-607ea62fd51d\" (UID: \"ba1e2c95-cd0f-43dc-89be-607ea62fd51d\") " Oct 30 13:03:21.252842 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:21.252813 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba1e2c95-cd0f-43dc-89be-607ea62fd51d-plugins-config-volume" pod "ba1e2c95-cd0f-43dc-89be-607ea62fd51d" (UID: "ba1e2c95-cd0f-43dc-89be-607ea62fd51d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:03:21.253178 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:21.253151 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba1e2c95-cd0f-43dc-89be-607ea62fd51d-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:03:21.254450 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:21.254426 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1e2c95-cd0f-43dc-89be-607ea62fd51d-kube-api-access-bqpr6" pod "ba1e2c95-cd0f-43dc-89be-607ea62fd51d" (UID: "ba1e2c95-cd0f-43dc-89be-607ea62fd51d"). InnerVolumeSpecName "kube-api-access-bqpr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:21.354763 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:21.354680 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqpr6\" (UniqueName: \"kubernetes.io/projected/ba1e2c95-cd0f-43dc-89be-607ea62fd51d-kube-api-access-bqpr6\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:03:21.477685 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:21.477650 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq"] Oct 30 13:03:21.482514 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:21.482479 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ecb3aa-6944494cb-82jdq"] Oct 30 13:03:23.091574 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:03:23.091544 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba1e2c95-cd0f-43dc-89be-607ea62fd51d" path="/var/lib/kubelet/pods/ba1e2c95-cd0f-43dc-89be-607ea62fd51d/volumes" Oct 30 13:03:23.599581 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:03:23.599548 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:03:53.602033 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:03:53.601954 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:04:21.942283 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:21.942247 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr"] Oct 30 13:04:21.942959 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:21.942905 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" podUID="c61a610b-08a7-46ce-b0d5-b2171147362c" containerName="maas-api" containerID="cri-o://2a77598c49ac829099e09c80072868d9536c8623a5bc20d7dbdf44f5a8dd7e58" gracePeriod=30 Oct 30 13:04:22.265105 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.265063 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" Oct 30 13:04:22.418196 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.418147 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c61a610b-08a7-46ce-b0d5-b2171147362c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61a610b-08a7-46ce-b0d5-b2171147362c-openshift-service-ca\") pod \"c61a610b-08a7-46ce-b0d5-b2171147362c\" (UID: \"c61a610b-08a7-46ce-b0d5-b2171147362c\") " Oct 30 13:04:22.418405 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.418291 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61a610b-08a7-46ce-b0d5-b2171147362c-kube-api-access-nh47k\" (UniqueName: \"kubernetes.io/projected/c61a610b-08a7-46ce-b0d5-b2171147362c-kube-api-access-nh47k\") pod \"c61a610b-08a7-46ce-b0d5-b2171147362c\" (UID: \"c61a610b-08a7-46ce-b0d5-b2171147362c\") " Oct 30 13:04:22.418405 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.418348 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c61a610b-08a7-46ce-b0d5-b2171147362c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c61a610b-08a7-46ce-b0d5-b2171147362c-metrics-certs\") pod \"c61a610b-08a7-46ce-b0d5-b2171147362c\" (UID: \"c61a610b-08a7-46ce-b0d5-b2171147362c\") " Oct 30 13:04:22.418532 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.418511 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c61a610b-08a7-46ce-b0d5-b2171147362c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c61a610b-08a7-46ce-b0d5-b2171147362c-maas-api-tls\") pod \"c61a610b-08a7-46ce-b0d5-b2171147362c\" (UID: \"c61a610b-08a7-46ce-b0d5-b2171147362c\") " Oct 30 13:04:22.418656 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.418618 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61a610b-08a7-46ce-b0d5-b2171147362c-openshift-service-ca" pod "c61a610b-08a7-46ce-b0d5-b2171147362c" (UID: "c61a610b-08a7-46ce-b0d5-b2171147362c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:22.419337 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.419308 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61a610b-08a7-46ce-b0d5-b2171147362c-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:22.420765 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.420740 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c61a610b-08a7-46ce-b0d5-b2171147362c-maas-api-tls" pod "c61a610b-08a7-46ce-b0d5-b2171147362c" (UID: "c61a610b-08a7-46ce-b0d5-b2171147362c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:22.420908 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.420769 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c61a610b-08a7-46ce-b0d5-b2171147362c-metrics-certs" pod "c61a610b-08a7-46ce-b0d5-b2171147362c" (UID: "c61a610b-08a7-46ce-b0d5-b2171147362c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:22.421307 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.421277 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61a610b-08a7-46ce-b0d5-b2171147362c-kube-api-access-nh47k" pod "c61a610b-08a7-46ce-b0d5-b2171147362c" (UID: "c61a610b-08a7-46ce-b0d5-b2171147362c"). InnerVolumeSpecName "kube-api-access-nh47k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:22.521440 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.521333 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh47k\" (UniqueName: \"kubernetes.io/projected/c61a610b-08a7-46ce-b0d5-b2171147362c-kube-api-access-nh47k\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:22.521440 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.521363 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c61a610b-08a7-46ce-b0d5-b2171147362c-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:22.521440 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.521373 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c61a610b-08a7-46ce-b0d5-b2171147362c-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:22.597571 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.597535 4468 generic.go:356] "Generic (PLEG): container finished" podID="c61a610b-08a7-46ce-b0d5-b2171147362c" containerID="2a77598c49ac829099e09c80072868d9536c8623a5bc20d7dbdf44f5a8dd7e58" exitCode=0 Oct 30 13:04:22.597755 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.597583 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" Oct 30 13:04:22.597755 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.597616 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" event={"ID":"c61a610b-08a7-46ce-b0d5-b2171147362c","Type":"ContainerDied","Data":"2a77598c49ac829099e09c80072868d9536c8623a5bc20d7dbdf44f5a8dd7e58"} Oct 30 13:04:22.597755 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.597672 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr" event={"ID":"c61a610b-08a7-46ce-b0d5-b2171147362c","Type":"ContainerDied","Data":"2510d1b9ab9a7d890b0590e23b9b000fe28deaab23a596253b10b8fa91a80370"} Oct 30 13:04:22.597755 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.597701 4468 scope.go:122] "RemoveContainer" containerID="2a77598c49ac829099e09c80072868d9536c8623a5bc20d7dbdf44f5a8dd7e58" Oct 30 13:04:22.610134 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.610111 4468 scope.go:122] "RemoveContainer" containerID="2a77598c49ac829099e09c80072868d9536c8623a5bc20d7dbdf44f5a8dd7e58" Oct 30 13:04:22.610411 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:04:22.610393 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a77598c49ac829099e09c80072868d9536c8623a5bc20d7dbdf44f5a8dd7e58\": container with ID starting with 2a77598c49ac829099e09c80072868d9536c8623a5bc20d7dbdf44f5a8dd7e58 not found: ID does not exist" containerID="2a77598c49ac829099e09c80072868d9536c8623a5bc20d7dbdf44f5a8dd7e58" Oct 30 13:04:22.610486 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.610418 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a77598c49ac829099e09c80072868d9536c8623a5bc20d7dbdf44f5a8dd7e58"} err="failed to get container status \"2a77598c49ac829099e09c80072868d9536c8623a5bc20d7dbdf44f5a8dd7e58\": rpc error: code = NotFound desc = could not find container \"2a77598c49ac829099e09c80072868d9536c8623a5bc20d7dbdf44f5a8dd7e58\": container with ID starting with 2a77598c49ac829099e09c80072868d9536c8623a5bc20d7dbdf44f5a8dd7e58 not found: ID does not exist" Oct 30 13:04:22.668936 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.668901 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr"] Oct 30 13:04:22.673934 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:22.673907 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0bda67-796f8666fd-w9drr"] Oct 30 13:04:23.093201 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:23.093161 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c61a610b-08a7-46ce-b0d5-b2171147362c" path="/var/lib/kubelet/pods/c61a610b-08a7-46ce-b0d5-b2171147362c/volumes" Oct 30 13:04:23.601047 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:04:23.601015 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:04:29.172310 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:29.172274 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf"] Oct 30 13:04:29.210325 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:29.210289 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2"] Oct 30 13:04:34.172734 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.172680 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf" podUID="52833cb7-6657-4225-9bf4-3532bcd3a15f" containerName="payload-processing" containerID="cri-o://67eca54fa8307cbe89997d3af4172c12b133ddb20ae22c5cfa4cf2285d227ec0" gracePeriod=25 Oct 30 13:04:34.212308 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.212036 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2" podUID="3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8" containerName="payload-pre-processing" containerID="cri-o://2bfba23f4da0441dc5efe0cdf54fdc93e5242a3c1fbc7ca13fd0e9945a20df8a" gracePeriod=25 Oct 30 13:04:34.495412 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.495381 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf" Oct 30 13:04:34.515945 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.515921 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52833cb7-6657-4225-9bf4-3532bcd3a15f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52833cb7-6657-4225-9bf4-3532bcd3a15f-plugins-config-volume\") pod \"52833cb7-6657-4225-9bf4-3532bcd3a15f\" (UID: \"52833cb7-6657-4225-9bf4-3532bcd3a15f\") " Oct 30 13:04:34.516102 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.516037 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52833cb7-6657-4225-9bf4-3532bcd3a15f-kube-api-access-d8czh\" (UniqueName: \"kubernetes.io/projected/52833cb7-6657-4225-9bf4-3532bcd3a15f-kube-api-access-d8czh\") pod \"52833cb7-6657-4225-9bf4-3532bcd3a15f\" (UID: \"52833cb7-6657-4225-9bf4-3532bcd3a15f\") " Oct 30 13:04:34.516597 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.516567 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52833cb7-6657-4225-9bf4-3532bcd3a15f-plugins-config-volume" pod "52833cb7-6657-4225-9bf4-3532bcd3a15f" (UID: "52833cb7-6657-4225-9bf4-3532bcd3a15f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:34.518429 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.518406 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52833cb7-6657-4225-9bf4-3532bcd3a15f-kube-api-access-d8czh" pod "52833cb7-6657-4225-9bf4-3532bcd3a15f" (UID: "52833cb7-6657-4225-9bf4-3532bcd3a15f"). InnerVolumeSpecName "kube-api-access-d8czh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:34.551513 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.551489 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2" Oct 30 13:04:34.617273 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.617238 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8-kube-api-access-fp944\" (UniqueName: \"kubernetes.io/projected/3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8-kube-api-access-fp944\") pod \"3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8\" (UID: \"3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8\") " Oct 30 13:04:34.617471 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.617381 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8-plugins-config-volume\") pod \"3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8\" (UID: \"3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8\") " Oct 30 13:04:34.617857 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.617790 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8-plugins-config-volume" pod "3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8" (UID: "3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:34.618130 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.618109 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:34.618230 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.618133 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8czh\" (UniqueName: \"kubernetes.io/projected/52833cb7-6657-4225-9bf4-3532bcd3a15f-kube-api-access-d8czh\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:34.618230 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.618143 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52833cb7-6657-4225-9bf4-3532bcd3a15f-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:34.620200 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.620175 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8-kube-api-access-fp944" pod "3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8" (UID: "3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8"). InnerVolumeSpecName "kube-api-access-fp944". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:34.709622 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.709584 4468 generic.go:356] "Generic (PLEG): container finished" podID="52833cb7-6657-4225-9bf4-3532bcd3a15f" containerID="67eca54fa8307cbe89997d3af4172c12b133ddb20ae22c5cfa4cf2285d227ec0" exitCode=0 Oct 30 13:04:34.709786 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.709626 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf" Oct 30 13:04:34.709786 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.709702 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf" event={"ID":"52833cb7-6657-4225-9bf4-3532bcd3a15f","Type":"ContainerDied","Data":"67eca54fa8307cbe89997d3af4172c12b133ddb20ae22c5cfa4cf2285d227ec0"} Oct 30 13:04:34.709786 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.709752 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf" event={"ID":"52833cb7-6657-4225-9bf4-3532bcd3a15f","Type":"ContainerDied","Data":"45397db40ae38195b4aa9a68949c515fd97ffee7696fab54dd389628ce02cfcd"} Oct 30 13:04:34.709786 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.709778 4468 scope.go:122] "RemoveContainer" containerID="67eca54fa8307cbe89997d3af4172c12b133ddb20ae22c5cfa4cf2285d227ec0" Oct 30 13:04:34.711749 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.711725 4468 generic.go:356] "Generic (PLEG): container finished" podID="3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8" containerID="2bfba23f4da0441dc5efe0cdf54fdc93e5242a3c1fbc7ca13fd0e9945a20df8a" exitCode=0 Oct 30 13:04:34.711885 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.711770 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2" Oct 30 13:04:34.711885 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.711777 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2" event={"ID":"3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8","Type":"ContainerDied","Data":"2bfba23f4da0441dc5efe0cdf54fdc93e5242a3c1fbc7ca13fd0e9945a20df8a"} Oct 30 13:04:34.711885 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.711821 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2" event={"ID":"3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8","Type":"ContainerDied","Data":"8f8de95dc324a14ff98dd03d10cefd3fb49879a3325e04e87a77181f9e405658"} Oct 30 13:04:34.718932 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.718876 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp944\" (UniqueName: \"kubernetes.io/projected/3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8-kube-api-access-fp944\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:34.724029 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.724011 4468 scope.go:122] "RemoveContainer" containerID="67eca54fa8307cbe89997d3af4172c12b133ddb20ae22c5cfa4cf2285d227ec0" Oct 30 13:04:34.724330 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:04:34.724303 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67eca54fa8307cbe89997d3af4172c12b133ddb20ae22c5cfa4cf2285d227ec0\": container with ID starting with 67eca54fa8307cbe89997d3af4172c12b133ddb20ae22c5cfa4cf2285d227ec0 not found: ID does not exist" containerID="67eca54fa8307cbe89997d3af4172c12b133ddb20ae22c5cfa4cf2285d227ec0" Oct 30 13:04:34.724411 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.724329 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67eca54fa8307cbe89997d3af4172c12b133ddb20ae22c5cfa4cf2285d227ec0"} err="failed to get container status \"67eca54fa8307cbe89997d3af4172c12b133ddb20ae22c5cfa4cf2285d227ec0\": rpc error: code = NotFound desc = could not find container \"67eca54fa8307cbe89997d3af4172c12b133ddb20ae22c5cfa4cf2285d227ec0\": container with ID starting with 67eca54fa8307cbe89997d3af4172c12b133ddb20ae22c5cfa4cf2285d227ec0 not found: ID does not exist" Oct 30 13:04:34.724411 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.724348 4468 scope.go:122] "RemoveContainer" containerID="2bfba23f4da0441dc5efe0cdf54fdc93e5242a3c1fbc7ca13fd0e9945a20df8a" Oct 30 13:04:34.735261 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.735245 4468 scope.go:122] "RemoveContainer" containerID="2bfba23f4da0441dc5efe0cdf54fdc93e5242a3c1fbc7ca13fd0e9945a20df8a" Oct 30 13:04:34.735500 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:04:34.735484 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bfba23f4da0441dc5efe0cdf54fdc93e5242a3c1fbc7ca13fd0e9945a20df8a\": container with ID starting with 2bfba23f4da0441dc5efe0cdf54fdc93e5242a3c1fbc7ca13fd0e9945a20df8a not found: ID does not exist" containerID="2bfba23f4da0441dc5efe0cdf54fdc93e5242a3c1fbc7ca13fd0e9945a20df8a" Oct 30 13:04:34.735564 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.735510 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfba23f4da0441dc5efe0cdf54fdc93e5242a3c1fbc7ca13fd0e9945a20df8a"} err="failed to get container status \"2bfba23f4da0441dc5efe0cdf54fdc93e5242a3c1fbc7ca13fd0e9945a20df8a\": rpc error: code = NotFound desc = could not find container \"2bfba23f4da0441dc5efe0cdf54fdc93e5242a3c1fbc7ca13fd0e9945a20df8a\": container with ID starting with 2bfba23f4da0441dc5efe0cdf54fdc93e5242a3c1fbc7ca13fd0e9945a20df8a not found: ID does not exist" Oct 30 13:04:34.743778 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:04:34.743757 4468 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52833cb7_6657_4225_9bf4_3532bcd3a15f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2f93b2_9ab8_4f16_afbb_6d5092ac2ba8.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:04:34.772954 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.772926 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf"] Oct 30 13:04:34.778446 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.778426 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6ad1da-87498f66b-wqcxf"] Oct 30 13:04:34.818827 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.818799 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2"] Oct 30 13:04:34.823803 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:34.823782 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6ad1da-74c77954b7-frpq2"] Oct 30 13:04:35.092854 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:35.092765 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8" path="/var/lib/kubelet/pods/3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8/volumes" Oct 30 13:04:35.093337 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:35.093314 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52833cb7-6657-4225-9bf4-3532bcd3a15f" path="/var/lib/kubelet/pods/52833cb7-6657-4225-9bf4-3532bcd3a15f/volumes" Oct 30 13:04:38.606990 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:38.606952 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv"] Oct 30 13:04:38.631737 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:38.631706 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs"] Oct 30 13:04:41.742651 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:41.742615 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf"] Oct 30 13:04:41.743182 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:41.742874 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" podUID="db33d91a-07fb-4c52-832e-0440bc4654af" containerName="istio-proxy" containerID="cri-o://c3fd1d62eecac0293c9ff5d09a3fb33834c62c4d6a05f3ca5609c2abf1b93114" gracePeriod=30 Oct 30 13:04:42.814201 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:42.814163 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g"] Oct 30 13:04:43.608211 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:43.608171 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv" podUID="6e29344f-41fb-4929-aee9-739cf692fa68" containerName="payload-processing" containerID="cri-o://615dcc27d96c7f6111bad8e313bca866ecde4b9b9ee31d78535b2c99de2238db" gracePeriod=25 Oct 30 13:04:43.637958 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:43.637908 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" podUID="9f68b80b-1983-4fc9-907f-20653341e98a" containerName="payload-pre-processing" containerID="cri-o://f3a38a1527d21d58ef88fd99cdee05e8f32f32c20e88846d5f6483a6edf25784" gracePeriod=25 Oct 30 13:04:43.812272 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:43.811563 4468 generic.go:356] "Generic (PLEG): container finished" podID="6e29344f-41fb-4929-aee9-739cf692fa68" containerID="615dcc27d96c7f6111bad8e313bca866ecde4b9b9ee31d78535b2c99de2238db" exitCode=0 Oct 30 13:04:43.812272 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:43.811721 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv" event={"ID":"6e29344f-41fb-4929-aee9-739cf692fa68","Type":"ContainerDied","Data":"615dcc27d96c7f6111bad8e313bca866ecde4b9b9ee31d78535b2c99de2238db"} Oct 30 13:04:43.901141 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:43.901118 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv" Oct 30 13:04:43.957649 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:43.957618 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e29344f-41fb-4929-aee9-739cf692fa68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e29344f-41fb-4929-aee9-739cf692fa68-plugins-config-volume\") pod \"6e29344f-41fb-4929-aee9-739cf692fa68\" (UID: \"6e29344f-41fb-4929-aee9-739cf692fa68\") " Oct 30 13:04:43.957807 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:43.957795 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e29344f-41fb-4929-aee9-739cf692fa68-kube-api-access-hm4xn\" (UniqueName: \"kubernetes.io/projected/6e29344f-41fb-4929-aee9-739cf692fa68-kube-api-access-hm4xn\") pod \"6e29344f-41fb-4929-aee9-739cf692fa68\" (UID: \"6e29344f-41fb-4929-aee9-739cf692fa68\") " Oct 30 13:04:43.958059 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:43.958033 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e29344f-41fb-4929-aee9-739cf692fa68-plugins-config-volume" pod "6e29344f-41fb-4929-aee9-739cf692fa68" (UID: "6e29344f-41fb-4929-aee9-739cf692fa68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:43.958455 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:43.958426 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e29344f-41fb-4929-aee9-739cf692fa68-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:43.959901 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:43.959877 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e29344f-41fb-4929-aee9-739cf692fa68-kube-api-access-hm4xn" pod "6e29344f-41fb-4929-aee9-739cf692fa68" (UID: "6e29344f-41fb-4929-aee9-739cf692fa68"). InnerVolumeSpecName "kube-api-access-hm4xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:44.060138 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:44.060110 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm4xn\" (UniqueName: \"kubernetes.io/projected/6e29344f-41fb-4929-aee9-739cf692fa68-kube-api-access-hm4xn\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:44.820376 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:44.820341 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv" Oct 30 13:04:44.820376 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:44.820360 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv" event={"ID":"6e29344f-41fb-4929-aee9-739cf692fa68","Type":"ContainerDied","Data":"77d4e6a59f3e60625690317684fb07204ec0add5dc840440ac26b08e85d3ca1b"} Oct 30 13:04:44.820600 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:44.820452 4468 scope.go:122] "RemoveContainer" containerID="615dcc27d96c7f6111bad8e313bca866ecde4b9b9ee31d78535b2c99de2238db" Oct 30 13:04:44.878399 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:44.878373 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv"] Oct 30 13:04:44.891738 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:44.891711 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b67f8-8564b6d7d6-j6hmv"] Oct 30 13:04:44.973112 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:44.970787 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg"] Oct 30 13:04:44.973112 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:44.971006 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" podUID="fca7e7d8-d36f-4378-8052-04a62530fff6" containerName="maas-api" containerID="cri-o://29aec5190929b65b8dd107d4127220f7c04142e01ca78c0359c78972a18e21dd" gracePeriod=30 Oct 30 13:04:45.092462 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.092342 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e29344f-41fb-4929-aee9-739cf692fa68" path="/var/lib/kubelet/pods/6e29344f-41fb-4929-aee9-739cf692fa68/volumes" Oct 30 13:04:45.313438 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.313414 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" Oct 30 13:04:45.396152 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.393854 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca7e7d8-d36f-4378-8052-04a62530fff6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca7e7d8-d36f-4378-8052-04a62530fff6-openshift-service-ca\") pod \"fca7e7d8-d36f-4378-8052-04a62530fff6\" (UID: \"fca7e7d8-d36f-4378-8052-04a62530fff6\") " Oct 30 13:04:45.396152 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.393981 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fca7e7d8-d36f-4378-8052-04a62530fff6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fca7e7d8-d36f-4378-8052-04a62530fff6-maas-api-tls\") pod \"fca7e7d8-d36f-4378-8052-04a62530fff6\" (UID: \"fca7e7d8-d36f-4378-8052-04a62530fff6\") " Oct 30 13:04:45.396152 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.394028 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca7e7d8-d36f-4378-8052-04a62530fff6-kube-api-access-nmkjd\" (UniqueName: \"kubernetes.io/projected/fca7e7d8-d36f-4378-8052-04a62530fff6-kube-api-access-nmkjd\") pod \"fca7e7d8-d36f-4378-8052-04a62530fff6\" (UID: \"fca7e7d8-d36f-4378-8052-04a62530fff6\") " Oct 30 13:04:45.396152 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.394112 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fca7e7d8-d36f-4378-8052-04a62530fff6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca7e7d8-d36f-4378-8052-04a62530fff6-metrics-certs\") pod \"fca7e7d8-d36f-4378-8052-04a62530fff6\" (UID: \"fca7e7d8-d36f-4378-8052-04a62530fff6\") " Oct 30 13:04:45.396152 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.395583 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca7e7d8-d36f-4378-8052-04a62530fff6-openshift-service-ca" pod "fca7e7d8-d36f-4378-8052-04a62530fff6" (UID: "fca7e7d8-d36f-4378-8052-04a62530fff6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:45.406641 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.406578 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca7e7d8-d36f-4378-8052-04a62530fff6-kube-api-access-nmkjd" pod "fca7e7d8-d36f-4378-8052-04a62530fff6" (UID: "fca7e7d8-d36f-4378-8052-04a62530fff6"). InnerVolumeSpecName "kube-api-access-nmkjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:45.441359 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.436952 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fca7e7d8-d36f-4378-8052-04a62530fff6-maas-api-tls" pod "fca7e7d8-d36f-4378-8052-04a62530fff6" (UID: "fca7e7d8-d36f-4378-8052-04a62530fff6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:45.441359 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.436978 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fca7e7d8-d36f-4378-8052-04a62530fff6-metrics-certs" pod "fca7e7d8-d36f-4378-8052-04a62530fff6" (UID: "fca7e7d8-d36f-4378-8052-04a62530fff6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:45.496881 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.496847 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca7e7d8-d36f-4378-8052-04a62530fff6-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:45.496881 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.496878 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fca7e7d8-d36f-4378-8052-04a62530fff6-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:45.497125 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.496893 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmkjd\" (UniqueName: \"kubernetes.io/projected/fca7e7d8-d36f-4378-8052-04a62530fff6-kube-api-access-nmkjd\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:45.497125 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.496912 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca7e7d8-d36f-4378-8052-04a62530fff6-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:45.829247 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.829205 4468 generic.go:356] "Generic (PLEG): container finished" podID="fca7e7d8-d36f-4378-8052-04a62530fff6" containerID="29aec5190929b65b8dd107d4127220f7c04142e01ca78c0359c78972a18e21dd" exitCode=0 Oct 30 13:04:45.829451 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.829249 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" Oct 30 13:04:45.829451 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.829282 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" event={"ID":"fca7e7d8-d36f-4378-8052-04a62530fff6","Type":"ContainerDied","Data":"29aec5190929b65b8dd107d4127220f7c04142e01ca78c0359c78972a18e21dd"} Oct 30 13:04:45.829451 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.829336 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg" event={"ID":"fca7e7d8-d36f-4378-8052-04a62530fff6","Type":"ContainerDied","Data":"cd33f193adce882466749d9e675890dfca0c071bc5b8bf312f9eb074fc5ac119"} Oct 30 13:04:45.829451 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.829358 4468 scope.go:122] "RemoveContainer" containerID="29aec5190929b65b8dd107d4127220f7c04142e01ca78c0359c78972a18e21dd" Oct 30 13:04:45.840287 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.840270 4468 scope.go:122] "RemoveContainer" containerID="29aec5190929b65b8dd107d4127220f7c04142e01ca78c0359c78972a18e21dd" Oct 30 13:04:45.840575 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:04:45.840549 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29aec5190929b65b8dd107d4127220f7c04142e01ca78c0359c78972a18e21dd\": container with ID starting with 29aec5190929b65b8dd107d4127220f7c04142e01ca78c0359c78972a18e21dd not found: ID does not exist" containerID="29aec5190929b65b8dd107d4127220f7c04142e01ca78c0359c78972a18e21dd" Oct 30 13:04:45.840650 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.840586 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29aec5190929b65b8dd107d4127220f7c04142e01ca78c0359c78972a18e21dd"} err="failed to get container status \"29aec5190929b65b8dd107d4127220f7c04142e01ca78c0359c78972a18e21dd\": rpc error: code = NotFound desc = could not find container \"29aec5190929b65b8dd107d4127220f7c04142e01ca78c0359c78972a18e21dd\": container with ID starting with 29aec5190929b65b8dd107d4127220f7c04142e01ca78c0359c78972a18e21dd not found: ID does not exist" Oct 30 13:04:45.891256 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.891227 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg"] Oct 30 13:04:45.896340 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:45.896318 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dbe123-67ff8dd546-5l6gg"] Oct 30 13:04:47.096222 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.096185 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca7e7d8-d36f-4378-8052-04a62530fff6" path="/var/lib/kubelet/pods/fca7e7d8-d36f-4378-8052-04a62530fff6/volumes" Oct 30 13:04:47.108023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.107996 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 13:04:47.222622 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.222582 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-workload-socket\") pod \"db33d91a-07fb-4c52-832e-0440bc4654af\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " Oct 30 13:04:47.222830 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.222811 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db33d91a-07fb-4c52-832e-0440bc4654af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db33d91a-07fb-4c52-832e-0440bc4654af-istio-podinfo\") pod \"db33d91a-07fb-4c52-832e-0440bc4654af\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " Oct 30 13:04:47.222911 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.222861 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-istio-envoy\") pod \"db33d91a-07fb-4c52-832e-0440bc4654af\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " Oct 30 13:04:47.222977 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.222912 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-workload-certs\") pod \"db33d91a-07fb-4c52-832e-0440bc4654af\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " Oct 30 13:04:47.222977 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.222947 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db33d91a-07fb-4c52-832e-0440bc4654af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db33d91a-07fb-4c52-832e-0440bc4654af-istiod-ca-cert\") pod \"db33d91a-07fb-4c52-832e-0440bc4654af\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " Oct 30 13:04:47.223138 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.222988 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db33d91a-07fb-4c52-832e-0440bc4654af-kube-api-access-vr9mt\" (UniqueName: \"kubernetes.io/projected/db33d91a-07fb-4c52-832e-0440bc4654af-kube-api-access-vr9mt\") pod \"db33d91a-07fb-4c52-832e-0440bc4654af\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " Oct 30 13:04:47.223138 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.223051 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db33d91a-07fb-4c52-832e-0440bc4654af-istio-token\" (UniqueName: \"kubernetes.io/projected/db33d91a-07fb-4c52-832e-0440bc4654af-istio-token\") pod \"db33d91a-07fb-4c52-832e-0440bc4654af\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " Oct 30 13:04:47.223268 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.223157 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-credential-socket\") pod \"db33d91a-07fb-4c52-832e-0440bc4654af\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " Oct 30 13:04:47.223268 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.223191 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-istio-data\") pod \"db33d91a-07fb-4c52-832e-0440bc4654af\" (UID: \"db33d91a-07fb-4c52-832e-0440bc4654af\") " Oct 30 13:04:47.223776 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.222904 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-workload-socket" pod "db33d91a-07fb-4c52-832e-0440bc4654af" (UID: "db33d91a-07fb-4c52-832e-0440bc4654af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:47.223776 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.223231 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-workload-certs" pod "db33d91a-07fb-4c52-832e-0440bc4654af" (UID: "db33d91a-07fb-4c52-832e-0440bc4654af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:47.224105 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.224051 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-credential-socket" pod "db33d91a-07fb-4c52-832e-0440bc4654af" (UID: "db33d91a-07fb-4c52-832e-0440bc4654af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:47.224105 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.224093 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-istio-data" pod "db33d91a-07fb-4c52-832e-0440bc4654af" (UID: "db33d91a-07fb-4c52-832e-0440bc4654af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:47.224401 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.224380 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db33d91a-07fb-4c52-832e-0440bc4654af-istiod-ca-cert" pod "db33d91a-07fb-4c52-832e-0440bc4654af" (UID: "db33d91a-07fb-4c52-832e-0440bc4654af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:47.226456 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.225959 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-istio-envoy" pod "db33d91a-07fb-4c52-832e-0440bc4654af" (UID: "db33d91a-07fb-4c52-832e-0440bc4654af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:47.227041 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.226940 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db33d91a-07fb-4c52-832e-0440bc4654af-istio-token" pod "db33d91a-07fb-4c52-832e-0440bc4654af" (UID: "db33d91a-07fb-4c52-832e-0440bc4654af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:47.227225 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.227189 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db33d91a-07fb-4c52-832e-0440bc4654af-istio-podinfo" pod "db33d91a-07fb-4c52-832e-0440bc4654af" (UID: "db33d91a-07fb-4c52-832e-0440bc4654af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:04:47.228262 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.228238 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db33d91a-07fb-4c52-832e-0440bc4654af-kube-api-access-vr9mt" pod "db33d91a-07fb-4c52-832e-0440bc4654af" (UID: "db33d91a-07fb-4c52-832e-0440bc4654af"). InnerVolumeSpecName "kube-api-access-vr9mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:47.325495 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.325417 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.325495 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.325458 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.325495 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.325473 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.325495 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.325487 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db33d91a-07fb-4c52-832e-0440bc4654af-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.325746 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.325502 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.325746 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.325515 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db33d91a-07fb-4c52-832e-0440bc4654af-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.325746 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.325529 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db33d91a-07fb-4c52-832e-0440bc4654af-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.325746 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.325542 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr9mt\" (UniqueName: \"kubernetes.io/projected/db33d91a-07fb-4c52-832e-0440bc4654af-kube-api-access-vr9mt\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.325746 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.325558 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db33d91a-07fb-4c52-832e-0440bc4654af-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.819104 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.818152 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g" podUID="36dd06a2-cbb4-43df-8411-f3b35e06e98c" containerName="payload-processing" containerID="cri-o://4bd9410d575837a4c5f95cbb47157222a2806d5996f1531e0d1c26ef37b0456c" gracePeriod=25 Oct 30 13:04:47.895786 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.894188 4468 generic.go:356] "Generic (PLEG): container finished" podID="db33d91a-07fb-4c52-832e-0440bc4654af" containerID="c3fd1d62eecac0293c9ff5d09a3fb33834c62c4d6a05f3ca5609c2abf1b93114" exitCode=0 Oct 30 13:04:47.895786 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.894443 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" Oct 30 13:04:47.895786 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.894489 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" event={"ID":"db33d91a-07fb-4c52-832e-0440bc4654af","Type":"ContainerDied","Data":"c3fd1d62eecac0293c9ff5d09a3fb33834c62c4d6a05f3ca5609c2abf1b93114"} Oct 30 13:04:47.895786 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.894551 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf" event={"ID":"db33d91a-07fb-4c52-832e-0440bc4654af","Type":"ContainerDied","Data":"d07ce4b56f8827b7cdce31e95bca80a93cc343b2fce1a423e0e660c2e122c9cf"} Oct 30 13:04:47.895786 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.894583 4468 scope.go:122] "RemoveContainer" containerID="c3fd1d62eecac0293c9ff5d09a3fb33834c62c4d6a05f3ca5609c2abf1b93114" Oct 30 13:04:47.940146 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.940116 4468 scope.go:122] "RemoveContainer" containerID="c27559fc290f639511e881027f3e747348dcd34f6f375bbf2ebb0009c8e11048" Oct 30 13:04:47.951226 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.951203 4468 scope.go:122] "RemoveContainer" containerID="c3fd1d62eecac0293c9ff5d09a3fb33834c62c4d6a05f3ca5609c2abf1b93114" Oct 30 13:04:47.951592 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:04:47.951564 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3fd1d62eecac0293c9ff5d09a3fb33834c62c4d6a05f3ca5609c2abf1b93114\": container with ID starting with c3fd1d62eecac0293c9ff5d09a3fb33834c62c4d6a05f3ca5609c2abf1b93114 not found: ID does not exist" containerID="c3fd1d62eecac0293c9ff5d09a3fb33834c62c4d6a05f3ca5609c2abf1b93114" Oct 30 13:04:47.951695 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.951602 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3fd1d62eecac0293c9ff5d09a3fb33834c62c4d6a05f3ca5609c2abf1b93114"} err="failed to get container status \"c3fd1d62eecac0293c9ff5d09a3fb33834c62c4d6a05f3ca5609c2abf1b93114\": rpc error: code = NotFound desc = could not find container \"c3fd1d62eecac0293c9ff5d09a3fb33834c62c4d6a05f3ca5609c2abf1b93114\": container with ID starting with c3fd1d62eecac0293c9ff5d09a3fb33834c62c4d6a05f3ca5609c2abf1b93114 not found: ID does not exist" Oct 30 13:04:47.951695 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.951627 4468 scope.go:122] "RemoveContainer" containerID="c27559fc290f639511e881027f3e747348dcd34f6f375bbf2ebb0009c8e11048" Oct 30 13:04:47.952058 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:04:47.952023 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c27559fc290f639511e881027f3e747348dcd34f6f375bbf2ebb0009c8e11048\": container with ID starting with c27559fc290f639511e881027f3e747348dcd34f6f375bbf2ebb0009c8e11048 not found: ID does not exist" containerID="c27559fc290f639511e881027f3e747348dcd34f6f375bbf2ebb0009c8e11048" Oct 30 13:04:47.952192 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.952102 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27559fc290f639511e881027f3e747348dcd34f6f375bbf2ebb0009c8e11048"} err="failed to get container status \"c27559fc290f639511e881027f3e747348dcd34f6f375bbf2ebb0009c8e11048\": rpc error: code = NotFound desc = could not find container \"c27559fc290f639511e881027f3e747348dcd34f6f375bbf2ebb0009c8e11048\": container with ID starting with c27559fc290f639511e881027f3e747348dcd34f6f375bbf2ebb0009c8e11048 not found: ID does not exist" Oct 30 13:04:47.957218 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.957195 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf"] Oct 30 13:04:47.971618 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:47.971587 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-6ad1da-openshift-default-54875f9cc7-btpwf"] Oct 30 13:04:48.130477 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:48.130453 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g" Oct 30 13:04:48.242179 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:48.242147 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36dd06a2-cbb4-43df-8411-f3b35e06e98c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36dd06a2-cbb4-43df-8411-f3b35e06e98c-plugins-config-volume\") pod \"36dd06a2-cbb4-43df-8411-f3b35e06e98c\" (UID: \"36dd06a2-cbb4-43df-8411-f3b35e06e98c\") " Oct 30 13:04:48.242366 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:48.242196 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36dd06a2-cbb4-43df-8411-f3b35e06e98c-kube-api-access-8spcq\" (UniqueName: \"kubernetes.io/projected/36dd06a2-cbb4-43df-8411-f3b35e06e98c-kube-api-access-8spcq\") pod \"36dd06a2-cbb4-43df-8411-f3b35e06e98c\" (UID: \"36dd06a2-cbb4-43df-8411-f3b35e06e98c\") " Oct 30 13:04:48.242515 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:48.242492 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36dd06a2-cbb4-43df-8411-f3b35e06e98c-plugins-config-volume" pod "36dd06a2-cbb4-43df-8411-f3b35e06e98c" (UID: "36dd06a2-cbb4-43df-8411-f3b35e06e98c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:48.242585 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:48.242575 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36dd06a2-cbb4-43df-8411-f3b35e06e98c-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:48.244526 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:48.244498 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36dd06a2-cbb4-43df-8411-f3b35e06e98c-kube-api-access-8spcq" pod "36dd06a2-cbb4-43df-8411-f3b35e06e98c" (UID: "36dd06a2-cbb4-43df-8411-f3b35e06e98c"). InnerVolumeSpecName "kube-api-access-8spcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:48.267726 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:48.267700 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8"] Oct 30 13:04:48.304751 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:48.304727 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c"] Oct 30 13:04:48.344202 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:48.344173 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8spcq\" (UniqueName: \"kubernetes.io/projected/36dd06a2-cbb4-43df-8411-f3b35e06e98c-kube-api-access-8spcq\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:04:48.920090 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:48.913791 4468 generic.go:356] "Generic (PLEG): container finished" podID="36dd06a2-cbb4-43df-8411-f3b35e06e98c" containerID="4bd9410d575837a4c5f95cbb47157222a2806d5996f1531e0d1c26ef37b0456c" exitCode=0 Oct 30 13:04:48.920090 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:48.913921 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g" event={"ID":"36dd06a2-cbb4-43df-8411-f3b35e06e98c","Type":"ContainerDied","Data":"4bd9410d575837a4c5f95cbb47157222a2806d5996f1531e0d1c26ef37b0456c"} Oct 30 13:04:48.920090 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:48.914006 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g" event={"ID":"36dd06a2-cbb4-43df-8411-f3b35e06e98c","Type":"ContainerDied","Data":"d2c0ad2b53b48db5b39ee589ea55eb1b285f114975d08412181837aa86c91fa5"} Oct 30 13:04:48.920090 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:48.914034 4468 scope.go:122] "RemoveContainer" containerID="4bd9410d575837a4c5f95cbb47157222a2806d5996f1531e0d1c26ef37b0456c" Oct 30 13:04:48.920090 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:48.914334 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g" Oct 30 13:04:48.928809 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:48.927885 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" podUID="9f68b80b-1983-4fc9-907f-20653341e98a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:04:48.949983 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:48.949928 4468 scope.go:122] "RemoveContainer" containerID="4bd9410d575837a4c5f95cbb47157222a2806d5996f1531e0d1c26ef37b0456c" Oct 30 13:04:48.952709 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:04:48.952667 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd9410d575837a4c5f95cbb47157222a2806d5996f1531e0d1c26ef37b0456c\": container with ID starting with 4bd9410d575837a4c5f95cbb47157222a2806d5996f1531e0d1c26ef37b0456c not found: ID does not exist" containerID="4bd9410d575837a4c5f95cbb47157222a2806d5996f1531e0d1c26ef37b0456c" Oct 30 13:04:48.952843 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:48.952708 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd9410d575837a4c5f95cbb47157222a2806d5996f1531e0d1c26ef37b0456c"} err="failed to get container status \"4bd9410d575837a4c5f95cbb47157222a2806d5996f1531e0d1c26ef37b0456c\": rpc error: code = NotFound desc = could not find container \"4bd9410d575837a4c5f95cbb47157222a2806d5996f1531e0d1c26ef37b0456c\": container with ID starting with 4bd9410d575837a4c5f95cbb47157222a2806d5996f1531e0d1c26ef37b0456c not found: ID does not exist" Oct 30 13:04:48.995181 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:48.995146 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g"] Oct 30 13:04:49.000626 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:49.000600 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-dbe123-768c9f5958-cbc4g"] Oct 30 13:04:49.092504 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:49.092473 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36dd06a2-cbb4-43df-8411-f3b35e06e98c" path="/var/lib/kubelet/pods/36dd06a2-cbb4-43df-8411-f3b35e06e98c/volumes" Oct 30 13:04:49.092991 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:49.092976 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db33d91a-07fb-4c52-832e-0440bc4654af" path="/var/lib/kubelet/pods/db33d91a-07fb-4c52-832e-0440bc4654af/volumes" Oct 30 13:04:52.789205 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:52.789166 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9"] Oct 30 13:04:52.834825 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:52.834784 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg"] Oct 30 13:04:53.268982 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:53.268942 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" podUID="416eb2ba-b5b4-4e25-a057-d870df27c1f7" containerName="payload-processing" containerID="cri-o://8414499e476289550a8fc4700a95df4b656895b3805ba68932d10402cba1545d" gracePeriod=25 Oct 30 13:04:53.304976 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:53.304948 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" podUID="add4979c-c091-4dea-9853-615756907859" containerName="payload-pre-processing" containerID="cri-o://0acc223ed0522d6bf07bb554761774d2b4fc6b6affa85511869c8ee557f31800" gracePeriod=25 Oct 30 13:04:53.602265 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:04:53.602184 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:04:57.790088 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:57.790015 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" podUID="ab60f224-4948-4a2b-abf2-116cb1827625" containerName="payload-processing" containerID="cri-o://0a58c2de2b8acde39654c3f72789c720e25918f63388c024e6a5ba4eb9f19354" gracePeriod=25 Oct 30 13:04:57.835599 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:57.835532 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" podUID="065b66a4-c80e-4dde-8bf6-4f14386e6d87" containerName="payload-pre-processing" containerID="cri-o://8da48b357841d8f1ea105eb6ccb75dafc58aa2b91555ebdea60ce07082755439" gracePeriod=25 Oct 30 13:04:58.724083 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:58.724046 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz"] Oct 30 13:04:58.738506 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:58.738478 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh"] Oct 30 13:04:58.916060 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:58.916021 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" podUID="9f68b80b-1983-4fc9-907f-20653341e98a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:04:59.319606 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:59.319572 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc"] Oct 30 13:04:59.319828 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:59.319802 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" podUID="e977ddbb-e36d-4056-b775-d91dbbb91881" containerName="maas-api" containerID="cri-o://1f0b1fb0f410a7fb4951338779f77bb5aafb40dabfe545db22523eb71cf5919b" gracePeriod=30 Oct 30 13:04:59.848853 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:59.848790 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" Oct 30 13:04:59.900623 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:59.900591 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e977ddbb-e36d-4056-b775-d91dbbb91881-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e977ddbb-e36d-4056-b775-d91dbbb91881-openshift-service-ca\") pod \"e977ddbb-e36d-4056-b775-d91dbbb91881\" (UID: \"e977ddbb-e36d-4056-b775-d91dbbb91881\") " Oct 30 13:04:59.900861 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:59.900707 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e977ddbb-e36d-4056-b775-d91dbbb91881-kube-api-access-bxhc6\" (UniqueName: \"kubernetes.io/projected/e977ddbb-e36d-4056-b775-d91dbbb91881-kube-api-access-bxhc6\") pod \"e977ddbb-e36d-4056-b775-d91dbbb91881\" (UID: \"e977ddbb-e36d-4056-b775-d91dbbb91881\") " Oct 30 13:04:59.900861 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:59.900826 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e977ddbb-e36d-4056-b775-d91dbbb91881-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e977ddbb-e36d-4056-b775-d91dbbb91881-maas-api-tls\") pod \"e977ddbb-e36d-4056-b775-d91dbbb91881\" (UID: \"e977ddbb-e36d-4056-b775-d91dbbb91881\") " Oct 30 13:04:59.900995 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:59.900944 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e977ddbb-e36d-4056-b775-d91dbbb91881-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e977ddbb-e36d-4056-b775-d91dbbb91881-metrics-certs\") pod \"e977ddbb-e36d-4056-b775-d91dbbb91881\" (UID: \"e977ddbb-e36d-4056-b775-d91dbbb91881\") " Oct 30 13:04:59.902002 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:59.901972 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e977ddbb-e36d-4056-b775-d91dbbb91881-openshift-service-ca" pod "e977ddbb-e36d-4056-b775-d91dbbb91881" (UID: "e977ddbb-e36d-4056-b775-d91dbbb91881"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:59.904377 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:59.904345 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e977ddbb-e36d-4056-b775-d91dbbb91881-kube-api-access-bxhc6" pod "e977ddbb-e36d-4056-b775-d91dbbb91881" (UID: "e977ddbb-e36d-4056-b775-d91dbbb91881"). InnerVolumeSpecName "kube-api-access-bxhc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:59.904492 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:59.904411 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e977ddbb-e36d-4056-b775-d91dbbb91881-metrics-certs" pod "e977ddbb-e36d-4056-b775-d91dbbb91881" (UID: "e977ddbb-e36d-4056-b775-d91dbbb91881"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:59.923205 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:04:59.923177 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e977ddbb-e36d-4056-b775-d91dbbb91881-maas-api-tls" pod "e977ddbb-e36d-4056-b775-d91dbbb91881" (UID: "e977ddbb-e36d-4056-b775-d91dbbb91881"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:00.002794 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:00.002767 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e977ddbb-e36d-4056-b775-d91dbbb91881-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:00.002794 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:00.002790 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e977ddbb-e36d-4056-b775-d91dbbb91881-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:00.002794 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:00.002800 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e977ddbb-e36d-4056-b775-d91dbbb91881-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:00.003044 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:00.002808 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxhc6\" (UniqueName: \"kubernetes.io/projected/e977ddbb-e36d-4056-b775-d91dbbb91881-kube-api-access-bxhc6\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:00.013515 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:00.013482 4468 generic.go:356] "Generic (PLEG): container finished" podID="e977ddbb-e36d-4056-b775-d91dbbb91881" containerID="1f0b1fb0f410a7fb4951338779f77bb5aafb40dabfe545db22523eb71cf5919b" exitCode=0 Oct 30 13:05:00.013659 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:00.013549 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" Oct 30 13:05:00.013733 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:00.013662 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" event={"ID":"e977ddbb-e36d-4056-b775-d91dbbb91881","Type":"ContainerDied","Data":"1f0b1fb0f410a7fb4951338779f77bb5aafb40dabfe545db22523eb71cf5919b"} Oct 30 13:05:00.013733 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:00.013706 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc" event={"ID":"e977ddbb-e36d-4056-b775-d91dbbb91881","Type":"ContainerDied","Data":"dffd58669961b36c3e50e50223790a98485eaeb98332424f1e0b965d52cc5802"} Oct 30 13:05:00.013844 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:00.013732 4468 scope.go:122] "RemoveContainer" containerID="1f0b1fb0f410a7fb4951338779f77bb5aafb40dabfe545db22523eb71cf5919b" Oct 30 13:05:00.026620 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:00.026596 4468 scope.go:122] "RemoveContainer" containerID="1f0b1fb0f410a7fb4951338779f77bb5aafb40dabfe545db22523eb71cf5919b" Oct 30 13:05:00.026906 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:05:00.026886 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f0b1fb0f410a7fb4951338779f77bb5aafb40dabfe545db22523eb71cf5919b\": container with ID starting with 1f0b1fb0f410a7fb4951338779f77bb5aafb40dabfe545db22523eb71cf5919b not found: ID does not exist" containerID="1f0b1fb0f410a7fb4951338779f77bb5aafb40dabfe545db22523eb71cf5919b" Oct 30 13:05:00.026974 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:00.026915 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f0b1fb0f410a7fb4951338779f77bb5aafb40dabfe545db22523eb71cf5919b"} err="failed to get container status \"1f0b1fb0f410a7fb4951338779f77bb5aafb40dabfe545db22523eb71cf5919b\": rpc error: code = NotFound desc = could not find container \"1f0b1fb0f410a7fb4951338779f77bb5aafb40dabfe545db22523eb71cf5919b\": container with ID starting with 1f0b1fb0f410a7fb4951338779f77bb5aafb40dabfe545db22523eb71cf5919b not found: ID does not exist" Oct 30 13:05:00.077330 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:00.077308 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc"] Oct 30 13:05:00.084154 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:00.084129 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-63f4e4-7cfd964cfc-7rbbc"] Oct 30 13:05:00.845949 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:00.845910 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" podUID="add4979c-c091-4dea-9853-615756907859" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:05:00.845949 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:00.845934 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" podUID="416eb2ba-b5b4-4e25-a057-d870df27c1f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:05:01.090952 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:01.090916 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e977ddbb-e36d-4056-b775-d91dbbb91881" path="/var/lib/kubelet/pods/e977ddbb-e36d-4056-b775-d91dbbb91881/volumes" Oct 30 13:05:03.925046 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:03.925015 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" Oct 30 13:05:04.045003 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.044966 4468 generic.go:356] "Generic (PLEG): container finished" podID="9f68b80b-1983-4fc9-907f-20653341e98a" containerID="f3a38a1527d21d58ef88fd99cdee05e8f32f32c20e88846d5f6483a6edf25784" exitCode=0 Oct 30 13:05:04.045188 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.045025 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" Oct 30 13:05:04.045188 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.045048 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" event={"ID":"9f68b80b-1983-4fc9-907f-20653341e98a","Type":"ContainerDied","Data":"f3a38a1527d21d58ef88fd99cdee05e8f32f32c20e88846d5f6483a6edf25784"} Oct 30 13:05:04.045188 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.045139 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs" event={"ID":"9f68b80b-1983-4fc9-907f-20653341e98a","Type":"ContainerDied","Data":"feb40f3b4d58d72e9f33a355b8ae7a80bf5754e4c1349fb81070f3d26edb35c1"} Oct 30 13:05:04.045188 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.045162 4468 scope.go:122] "RemoveContainer" containerID="f3a38a1527d21d58ef88fd99cdee05e8f32f32c20e88846d5f6483a6edf25784" Oct 30 13:05:04.055610 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.055588 4468 scope.go:122] "RemoveContainer" containerID="f3a38a1527d21d58ef88fd99cdee05e8f32f32c20e88846d5f6483a6edf25784" Oct 30 13:05:04.055901 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:05:04.055879 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a38a1527d21d58ef88fd99cdee05e8f32f32c20e88846d5f6483a6edf25784\": container with ID starting with f3a38a1527d21d58ef88fd99cdee05e8f32f32c20e88846d5f6483a6edf25784 not found: ID does not exist" containerID="f3a38a1527d21d58ef88fd99cdee05e8f32f32c20e88846d5f6483a6edf25784" Oct 30 13:05:04.055981 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.055907 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a38a1527d21d58ef88fd99cdee05e8f32f32c20e88846d5f6483a6edf25784"} err="failed to get container status \"f3a38a1527d21d58ef88fd99cdee05e8f32f32c20e88846d5f6483a6edf25784\": rpc error: code = NotFound desc = could not find container \"f3a38a1527d21d58ef88fd99cdee05e8f32f32c20e88846d5f6483a6edf25784\": container with ID starting with f3a38a1527d21d58ef88fd99cdee05e8f32f32c20e88846d5f6483a6edf25784 not found: ID does not exist" Oct 30 13:05:04.057871 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.057854 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f68b80b-1983-4fc9-907f-20653341e98a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f68b80b-1983-4fc9-907f-20653341e98a-plugins-config-volume\") pod \"9f68b80b-1983-4fc9-907f-20653341e98a\" (UID: \"9f68b80b-1983-4fc9-907f-20653341e98a\") " Oct 30 13:05:04.057936 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.057917 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f68b80b-1983-4fc9-907f-20653341e98a-kube-api-access-76mr5\" (UniqueName: \"kubernetes.io/projected/9f68b80b-1983-4fc9-907f-20653341e98a-kube-api-access-76mr5\") pod \"9f68b80b-1983-4fc9-907f-20653341e98a\" (UID: \"9f68b80b-1983-4fc9-907f-20653341e98a\") " Oct 30 13:05:04.058210 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.058191 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f68b80b-1983-4fc9-907f-20653341e98a-plugins-config-volume" pod "9f68b80b-1983-4fc9-907f-20653341e98a" (UID: "9f68b80b-1983-4fc9-907f-20653341e98a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:04.058466 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.058453 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f68b80b-1983-4fc9-907f-20653341e98a-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:04.059920 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.059901 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f68b80b-1983-4fc9-907f-20653341e98a-kube-api-access-76mr5" pod "9f68b80b-1983-4fc9-907f-20653341e98a" (UID: "9f68b80b-1983-4fc9-907f-20653341e98a"). InnerVolumeSpecName "kube-api-access-76mr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:04.159410 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.159382 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76mr5\" (UniqueName: \"kubernetes.io/projected/9f68b80b-1983-4fc9-907f-20653341e98a-kube-api-access-76mr5\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:04.348767 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.348736 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp"] Oct 30 13:05:04.383670 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.383644 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7"] Oct 30 13:05:04.445844 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.445758 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" podUID="065b66a4-c80e-4dde-8bf6-4f14386e6d87" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:05:04.446358 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.446328 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" podUID="ab60f224-4948-4a2b-abf2-116cb1827625" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:05:04.482799 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.482763 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs"] Oct 30 13:05:04.490893 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.490866 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b67f8-86cc7b4f9-8lvcs"] Oct 30 13:05:04.831466 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.831364 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt"] Oct 30 13:05:04.831660 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:04.831631 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" podUID="b60e79ec-6b0e-4725-81f3-d02c72fc8efb" containerName="istio-proxy" containerID="cri-o://0e5f3ecb5eb7880bc204bb9e9feaa5f9f3f6ae730510603e65e48f86032af4f5" gracePeriod=30 Oct 30 13:05:05.092004 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:05.091927 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f68b80b-1983-4fc9-907f-20653341e98a" path="/var/lib/kubelet/pods/9f68b80b-1983-4fc9-907f-20653341e98a/volumes" Oct 30 13:05:09.349473 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:09.349432 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" podUID="42777387-98e9-4718-aa0e-dcdcb64b7a19" containerName="payload-processing" containerID="cri-o://af1a9bc7dbe714cc2d93960e8d320b235695947cda49bae0e6b48c038d4b6401" gracePeriod=25 Oct 30 13:05:09.384175 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:09.384118 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" podUID="2dbcae46-3576-48d9-9691-50ddfaefd50e" containerName="payload-pre-processing" containerID="cri-o://23e5ef21dca0e9e75d2ce2fe9ebd7d8406c3163ad3d33f32eab5d5904f7c04f0" gracePeriod=25 Oct 30 13:05:10.108971 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.108940 4468 generic.go:356] "Generic (PLEG): container finished" podID="b60e79ec-6b0e-4725-81f3-d02c72fc8efb" containerID="0e5f3ecb5eb7880bc204bb9e9feaa5f9f3f6ae730510603e65e48f86032af4f5" exitCode=0 Oct 30 13:05:10.109144 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.109033 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" event={"ID":"b60e79ec-6b0e-4725-81f3-d02c72fc8efb","Type":"ContainerDied","Data":"0e5f3ecb5eb7880bc204bb9e9feaa5f9f3f6ae730510603e65e48f86032af4f5"} Oct 30 13:05:10.167588 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.167560 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 13:05:10.242657 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.242626 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istiod-ca-cert\") pod \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " Oct 30 13:05:10.242815 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.242674 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-podinfo\") pod \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " Oct 30 13:05:10.242815 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.242749 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-workload-socket\") pod \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " Oct 30 13:05:10.243033 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.242914 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-credential-socket\") pod \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " Oct 30 13:05:10.243033 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.242955 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-token\" (UniqueName: \"kubernetes.io/projected/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-token\") pod \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " Oct 30 13:05:10.243176 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.243034 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-data\") pod \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " Oct 30 13:05:10.243176 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.243054 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istiod-ca-cert" pod "b60e79ec-6b0e-4725-81f3-d02c72fc8efb" (UID: "b60e79ec-6b0e-4725-81f3-d02c72fc8efb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:10.243176 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.243106 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-kube-api-access-rrt2p\" (UniqueName: \"kubernetes.io/projected/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-kube-api-access-rrt2p\") pod \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " Oct 30 13:05:10.243360 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.243283 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-envoy\") pod \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " Oct 30 13:05:10.243360 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.243327 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-workload-certs\") pod \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\" (UID: \"b60e79ec-6b0e-4725-81f3-d02c72fc8efb\") " Oct 30 13:05:10.244502 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.243643 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-data" pod "b60e79ec-6b0e-4725-81f3-d02c72fc8efb" (UID: "b60e79ec-6b0e-4725-81f3-d02c72fc8efb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:10.244502 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.243876 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-credential-socket" pod "b60e79ec-6b0e-4725-81f3-d02c72fc8efb" (UID: "b60e79ec-6b0e-4725-81f3-d02c72fc8efb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:10.244502 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.243891 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-workload-certs" pod "b60e79ec-6b0e-4725-81f3-d02c72fc8efb" (UID: "b60e79ec-6b0e-4725-81f3-d02c72fc8efb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:10.244502 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.244190 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-workload-socket" pod "b60e79ec-6b0e-4725-81f3-d02c72fc8efb" (UID: "b60e79ec-6b0e-4725-81f3-d02c72fc8efb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:10.244502 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.244416 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:10.244502 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.244435 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:10.244502 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.244449 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:10.244502 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.244462 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:10.244502 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.244473 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:10.246118 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.246062 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-podinfo" pod "b60e79ec-6b0e-4725-81f3-d02c72fc8efb" (UID: "b60e79ec-6b0e-4725-81f3-d02c72fc8efb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:05:10.247126 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.247101 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-envoy" pod "b60e79ec-6b0e-4725-81f3-d02c72fc8efb" (UID: "b60e79ec-6b0e-4725-81f3-d02c72fc8efb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:10.247254 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.247146 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-kube-api-access-rrt2p" pod "b60e79ec-6b0e-4725-81f3-d02c72fc8efb" (UID: "b60e79ec-6b0e-4725-81f3-d02c72fc8efb"). InnerVolumeSpecName "kube-api-access-rrt2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:10.248672 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.248648 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-token" pod "b60e79ec-6b0e-4725-81f3-d02c72fc8efb" (UID: "b60e79ec-6b0e-4725-81f3-d02c72fc8efb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:10.345676 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.345643 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:10.345676 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.345672 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrt2p\" (UniqueName: \"kubernetes.io/projected/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-kube-api-access-rrt2p\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:10.345676 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.345682 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:10.345903 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.345691 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b60e79ec-6b0e-4725-81f3-d02c72fc8efb-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:10.846566 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.846525 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" podUID="add4979c-c091-4dea-9853-615756907859" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:05:10.846965 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:10.846543 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" podUID="416eb2ba-b5b4-4e25-a057-d870df27c1f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:05:11.117495 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:11.117471 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" Oct 30 13:05:11.117631 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:11.117509 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt" event={"ID":"b60e79ec-6b0e-4725-81f3-d02c72fc8efb","Type":"ContainerDied","Data":"8b81536df49c05389a635023394682cda9241741585b9d92847de617f19d1c0b"} Oct 30 13:05:11.117631 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:11.117575 4468 scope.go:122] "RemoveContainer" containerID="0e5f3ecb5eb7880bc204bb9e9feaa5f9f3f6ae730510603e65e48f86032af4f5" Oct 30 13:05:11.190776 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:11.190749 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt"] Oct 30 13:05:11.195284 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:11.195257 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kxnqt"] Oct 30 13:05:13.091518 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:13.091486 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b60e79ec-6b0e-4725-81f3-d02c72fc8efb" path="/var/lib/kubelet/pods/b60e79ec-6b0e-4725-81f3-d02c72fc8efb/volumes" Oct 30 13:05:13.590801 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:13.590777 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" Oct 30 13:05:13.628110 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:13.628065 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" Oct 30 13:05:13.702361 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:13.702325 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416eb2ba-b5b4-4e25-a057-d870df27c1f7-kube-api-access-fqmqg\" (UniqueName: \"kubernetes.io/projected/416eb2ba-b5b4-4e25-a057-d870df27c1f7-kube-api-access-fqmqg\") pod \"416eb2ba-b5b4-4e25-a057-d870df27c1f7\" (UID: \"416eb2ba-b5b4-4e25-a057-d870df27c1f7\") " Oct 30 13:05:13.702670 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:13.702652 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/416eb2ba-b5b4-4e25-a057-d870df27c1f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416eb2ba-b5b4-4e25-a057-d870df27c1f7-plugins-config-volume\") pod \"416eb2ba-b5b4-4e25-a057-d870df27c1f7\" (UID: \"416eb2ba-b5b4-4e25-a057-d870df27c1f7\") " Oct 30 13:05:13.703245 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:13.703216 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416eb2ba-b5b4-4e25-a057-d870df27c1f7-plugins-config-volume" pod "416eb2ba-b5b4-4e25-a057-d870df27c1f7" (UID: "416eb2ba-b5b4-4e25-a057-d870df27c1f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:13.703507 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:13.703483 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416eb2ba-b5b4-4e25-a057-d870df27c1f7-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:13.705209 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:13.705180 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416eb2ba-b5b4-4e25-a057-d870df27c1f7-kube-api-access-fqmqg" pod "416eb2ba-b5b4-4e25-a057-d870df27c1f7" (UID: "416eb2ba-b5b4-4e25-a057-d870df27c1f7"). InnerVolumeSpecName "kube-api-access-fqmqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:13.748248 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:13.748220 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" containerID="cri-o://99769e6d7b4aa20d868c87df93106ccb2e498389923bf38f94c73be30e5064f0" gracePeriod=45 Oct 30 13:05:13.761233 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:13.761197 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" containerID="cri-o://668f6202dfac84c3c3345cf7bb340c8c0cb6df8f2bf89574e00bf3d8721ef881" gracePeriod=45 Oct 30 13:05:13.804240 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:13.804205 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add4979c-c091-4dea-9853-615756907859-kube-api-access-4snpb\" (UniqueName: \"kubernetes.io/projected/add4979c-c091-4dea-9853-615756907859-kube-api-access-4snpb\") pod \"add4979c-c091-4dea-9853-615756907859\" (UID: \"add4979c-c091-4dea-9853-615756907859\") " Oct 30 13:05:13.804352 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:13.804263 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add4979c-c091-4dea-9853-615756907859-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add4979c-c091-4dea-9853-615756907859-plugins-config-volume\") pod \"add4979c-c091-4dea-9853-615756907859\" (UID: \"add4979c-c091-4dea-9853-615756907859\") " Oct 30 13:05:13.804682 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:13.804658 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add4979c-c091-4dea-9853-615756907859-plugins-config-volume" pod "add4979c-c091-4dea-9853-615756907859" (UID: "add4979c-c091-4dea-9853-615756907859"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:13.804844 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:13.804823 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqmqg\" (UniqueName: \"kubernetes.io/projected/416eb2ba-b5b4-4e25-a057-d870df27c1f7-kube-api-access-fqmqg\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:13.804891 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:13.804846 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add4979c-c091-4dea-9853-615756907859-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:13.806200 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:13.806170 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add4979c-c091-4dea-9853-615756907859-kube-api-access-4snpb" pod "add4979c-c091-4dea-9853-615756907859" (UID: "add4979c-c091-4dea-9853-615756907859"). InnerVolumeSpecName "kube-api-access-4snpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:13.907064 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:13.907021 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4snpb\" (UniqueName: \"kubernetes.io/projected/add4979c-c091-4dea-9853-615756907859-kube-api-access-4snpb\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.031166 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.031133 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 13:05:14.050863 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.050838 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 13:05:14.149146 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.149056 4468 generic.go:356] "Generic (PLEG): container finished" podID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerID="668f6202dfac84c3c3345cf7bb340c8c0cb6df8f2bf89574e00bf3d8721ef881" exitCode=0 Oct 30 13:05:14.149146 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.149148 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" Oct 30 13:05:14.149671 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.149165 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" event={"ID":"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1","Type":"ContainerDied","Data":"668f6202dfac84c3c3345cf7bb340c8c0cb6df8f2bf89574e00bf3d8721ef881"} Oct 30 13:05:14.149671 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.149217 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" event={"ID":"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1","Type":"ContainerDied","Data":"c72f400583e05574e20c661a55377ffeea535d247006881a184cfdb851ccee49"} Oct 30 13:05:14.149671 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.149242 4468 scope.go:122] "RemoveContainer" containerID="668f6202dfac84c3c3345cf7bb340c8c0cb6df8f2bf89574e00bf3d8721ef881" Oct 30 13:05:14.151307 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.151285 4468 generic.go:356] "Generic (PLEG): container finished" podID="416eb2ba-b5b4-4e25-a057-d870df27c1f7" containerID="8414499e476289550a8fc4700a95df4b656895b3805ba68932d10402cba1545d" exitCode=0 Oct 30 13:05:14.151422 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.151377 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" event={"ID":"416eb2ba-b5b4-4e25-a057-d870df27c1f7","Type":"ContainerDied","Data":"8414499e476289550a8fc4700a95df4b656895b3805ba68932d10402cba1545d"} Oct 30 13:05:14.151487 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.151461 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" event={"ID":"416eb2ba-b5b4-4e25-a057-d870df27c1f7","Type":"ContainerDied","Data":"fc8dc2d0d7d4437cf976af28cd8e66b90d6658368a830316ec7afb199163905f"} Oct 30 13:05:14.151989 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.151368 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8" Oct 30 13:05:14.153531 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.153508 4468 generic.go:356] "Generic (PLEG): container finished" podID="d273864b-7bab-4471-8531-3855be5b6182" containerID="99769e6d7b4aa20d868c87df93106ccb2e498389923bf38f94c73be30e5064f0" exitCode=0 Oct 30 13:05:14.153757 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.153611 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" Oct 30 13:05:14.153882 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.153629 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" event={"ID":"d273864b-7bab-4471-8531-3855be5b6182","Type":"ContainerDied","Data":"99769e6d7b4aa20d868c87df93106ccb2e498389923bf38f94c73be30e5064f0"} Oct 30 13:05:14.153968 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.153904 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" event={"ID":"d273864b-7bab-4471-8531-3855be5b6182","Type":"ContainerDied","Data":"1b6ce39e5bcd272057ae0e234ec28adbc962b7a7b04b49aca07d1ba3e9628737"} Oct 30 13:05:14.155951 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.155921 4468 generic.go:356] "Generic (PLEG): container finished" podID="add4979c-c091-4dea-9853-615756907859" containerID="0acc223ed0522d6bf07bb554761774d2b4fc6b6affa85511869c8ee557f31800" exitCode=0 Oct 30 13:05:14.156052 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.156012 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" event={"ID":"add4979c-c091-4dea-9853-615756907859","Type":"ContainerDied","Data":"0acc223ed0522d6bf07bb554761774d2b4fc6b6affa85511869c8ee557f31800"} Oct 30 13:05:14.156288 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.156269 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" Oct 30 13:05:14.161164 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.161134 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c" event={"ID":"add4979c-c091-4dea-9853-615756907859","Type":"ContainerDied","Data":"765c7692ea2a92fdb14899acfcbe9e25cfeb7d5ff0feb45e754f6a5a89bec90c"} Oct 30 13:05:14.169807 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.169785 4468 scope.go:122] "RemoveContainer" containerID="668f6202dfac84c3c3345cf7bb340c8c0cb6df8f2bf89574e00bf3d8721ef881" Oct 30 13:05:14.170734 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:05:14.170709 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"668f6202dfac84c3c3345cf7bb340c8c0cb6df8f2bf89574e00bf3d8721ef881\": container with ID starting with 668f6202dfac84c3c3345cf7bb340c8c0cb6df8f2bf89574e00bf3d8721ef881 not found: ID does not exist" containerID="668f6202dfac84c3c3345cf7bb340c8c0cb6df8f2bf89574e00bf3d8721ef881" Oct 30 13:05:14.170848 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.170743 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"668f6202dfac84c3c3345cf7bb340c8c0cb6df8f2bf89574e00bf3d8721ef881"} err="failed to get container status \"668f6202dfac84c3c3345cf7bb340c8c0cb6df8f2bf89574e00bf3d8721ef881\": rpc error: code = NotFound desc = could not find container \"668f6202dfac84c3c3345cf7bb340c8c0cb6df8f2bf89574e00bf3d8721ef881\": container with ID starting with 668f6202dfac84c3c3345cf7bb340c8c0cb6df8f2bf89574e00bf3d8721ef881 not found: ID does not exist" Oct 30 13:05:14.170848 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.170791 4468 scope.go:122] "RemoveContainer" containerID="8414499e476289550a8fc4700a95df4b656895b3805ba68932d10402cba1545d" Oct 30 13:05:14.181449 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.181430 4468 scope.go:122] "RemoveContainer" containerID="8414499e476289550a8fc4700a95df4b656895b3805ba68932d10402cba1545d" Oct 30 13:05:14.181784 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:05:14.181759 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8414499e476289550a8fc4700a95df4b656895b3805ba68932d10402cba1545d\": container with ID starting with 8414499e476289550a8fc4700a95df4b656895b3805ba68932d10402cba1545d not found: ID does not exist" containerID="8414499e476289550a8fc4700a95df4b656895b3805ba68932d10402cba1545d" Oct 30 13:05:14.181850 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.181793 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8414499e476289550a8fc4700a95df4b656895b3805ba68932d10402cba1545d"} err="failed to get container status \"8414499e476289550a8fc4700a95df4b656895b3805ba68932d10402cba1545d\": rpc error: code = NotFound desc = could not find container \"8414499e476289550a8fc4700a95df4b656895b3805ba68932d10402cba1545d\": container with ID starting with 8414499e476289550a8fc4700a95df4b656895b3805ba68932d10402cba1545d not found: ID does not exist" Oct 30 13:05:14.181850 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.181815 4468 scope.go:122] "RemoveContainer" containerID="99769e6d7b4aa20d868c87df93106ccb2e498389923bf38f94c73be30e5064f0" Oct 30 13:05:14.193552 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.193520 4468 scope.go:122] "RemoveContainer" containerID="99769e6d7b4aa20d868c87df93106ccb2e498389923bf38f94c73be30e5064f0" Oct 30 13:05:14.193837 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:05:14.193812 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99769e6d7b4aa20d868c87df93106ccb2e498389923bf38f94c73be30e5064f0\": container with ID starting with 99769e6d7b4aa20d868c87df93106ccb2e498389923bf38f94c73be30e5064f0 not found: ID does not exist" containerID="99769e6d7b4aa20d868c87df93106ccb2e498389923bf38f94c73be30e5064f0" Oct 30 13:05:14.193910 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.193847 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99769e6d7b4aa20d868c87df93106ccb2e498389923bf38f94c73be30e5064f0"} err="failed to get container status \"99769e6d7b4aa20d868c87df93106ccb2e498389923bf38f94c73be30e5064f0\": rpc error: code = NotFound desc = could not find container \"99769e6d7b4aa20d868c87df93106ccb2e498389923bf38f94c73be30e5064f0\": container with ID starting with 99769e6d7b4aa20d868c87df93106ccb2e498389923bf38f94c73be30e5064f0 not found: ID does not exist" Oct 30 13:05:14.193910 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.193868 4468 scope.go:122] "RemoveContainer" containerID="0acc223ed0522d6bf07bb554761774d2b4fc6b6affa85511869c8ee557f31800" Oct 30 13:05:14.203388 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.203372 4468 scope.go:122] "RemoveContainer" containerID="0acc223ed0522d6bf07bb554761774d2b4fc6b6affa85511869c8ee557f31800" Oct 30 13:05:14.203657 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:05:14.203635 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0acc223ed0522d6bf07bb554761774d2b4fc6b6affa85511869c8ee557f31800\": container with ID starting with 0acc223ed0522d6bf07bb554761774d2b4fc6b6affa85511869c8ee557f31800 not found: ID does not exist" containerID="0acc223ed0522d6bf07bb554761774d2b4fc6b6affa85511869c8ee557f31800" Oct 30 13:05:14.203772 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.203660 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acc223ed0522d6bf07bb554761774d2b4fc6b6affa85511869c8ee557f31800"} err="failed to get container status \"0acc223ed0522d6bf07bb554761774d2b4fc6b6affa85511869c8ee557f31800\": rpc error: code = NotFound desc = could not find container \"0acc223ed0522d6bf07bb554761774d2b4fc6b6affa85511869c8ee557f31800\": container with ID starting with 0acc223ed0522d6bf07bb554761774d2b4fc6b6affa85511869c8ee557f31800 not found: ID does not exist" Oct 30 13:05:14.211257 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.211235 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-dshm\") pod \"d273864b-7bab-4471-8531-3855be5b6182\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " Oct 30 13:05:14.211359 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.211309 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-home\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-home\") pod \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " Oct 30 13:05:14.211425 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.211412 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-dshm\") pod \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " Oct 30 13:05:14.211485 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.211454 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-tmp-dir\") pod \"d273864b-7bab-4471-8531-3855be5b6182\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " Oct 30 13:05:14.211885 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.211549 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-model-cache\") pod \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " Oct 30 13:05:14.211885 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.211586 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-home\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-home\") pod \"d273864b-7bab-4471-8531-3855be5b6182\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " Oct 30 13:05:14.211885 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.211599 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-home" pod "ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" (UID: "ea95fef4-f1e3-4967-9c5f-b6133ae05eb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:14.211885 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.211620 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-tls-certs\") pod \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " Oct 30 13:05:14.211885 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.211702 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-tmp-dir\") pod \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " Oct 30 13:05:14.211885 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.211750 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-kube-api-access-5c6cr\" (UniqueName: \"kubernetes.io/projected/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-kube-api-access-5c6cr\") pod \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\" (UID: \"ea95fef4-f1e3-4967-9c5f-b6133ae05eb1\") " Oct 30 13:05:14.211885 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.211790 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d273864b-7bab-4471-8531-3855be5b6182-tls-certs\" (UniqueName: \"kubernetes.io/secret/d273864b-7bab-4471-8531-3855be5b6182-tls-certs\") pod \"d273864b-7bab-4471-8531-3855be5b6182\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " Oct 30 13:05:14.211885 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.211793 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-model-cache" pod "ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" (UID: "ea95fef4-f1e3-4967-9c5f-b6133ae05eb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:14.211885 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.211806 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-tmp-dir" pod "d273864b-7bab-4471-8531-3855be5b6182" (UID: "d273864b-7bab-4471-8531-3855be5b6182"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:14.211885 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.211823 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d273864b-7bab-4471-8531-3855be5b6182-kube-api-access-zv5tq\" (UniqueName: \"kubernetes.io/projected/d273864b-7bab-4471-8531-3855be5b6182-kube-api-access-zv5tq\") pod \"d273864b-7bab-4471-8531-3855be5b6182\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " Oct 30 13:05:14.211885 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.211860 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-model-cache\") pod \"d273864b-7bab-4471-8531-3855be5b6182\" (UID: \"d273864b-7bab-4471-8531-3855be5b6182\") " Oct 30 13:05:14.212618 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.212357 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-home" pod "d273864b-7bab-4471-8531-3855be5b6182" (UID: "d273864b-7bab-4471-8531-3855be5b6182"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:14.213057 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.212443 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-tmp-dir" pod "ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" (UID: "ea95fef4-f1e3-4967-9c5f-b6133ae05eb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:14.213057 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.212989 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-model-cache" pod "d273864b-7bab-4471-8531-3855be5b6182" (UID: "d273864b-7bab-4471-8531-3855be5b6182"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:14.213251 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.212449 4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-tmp-dir\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.213251 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.213098 4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-model-cache\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.213251 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.213115 4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-home\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.214166 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.214137 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-dshm" pod "d273864b-7bab-4471-8531-3855be5b6182" (UID: "d273864b-7bab-4471-8531-3855be5b6182"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:14.214699 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.214669 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-tls-certs" pod "ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" (UID: "ea95fef4-f1e3-4967-9c5f-b6133ae05eb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:14.214776 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.214699 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-dshm" pod "ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" (UID: "ea95fef4-f1e3-4967-9c5f-b6133ae05eb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:14.214900 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.214878 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d273864b-7bab-4471-8531-3855be5b6182-tls-certs" pod "d273864b-7bab-4471-8531-3855be5b6182" (UID: "d273864b-7bab-4471-8531-3855be5b6182"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:14.215277 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.215257 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d273864b-7bab-4471-8531-3855be5b6182-kube-api-access-zv5tq" pod "d273864b-7bab-4471-8531-3855be5b6182" (UID: "d273864b-7bab-4471-8531-3855be5b6182"). InnerVolumeSpecName "kube-api-access-zv5tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:14.215351 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.215322 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-kube-api-access-5c6cr" pod "ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" (UID: "ea95fef4-f1e3-4967-9c5f-b6133ae05eb1"). InnerVolumeSpecName "kube-api-access-5c6cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:14.260541 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.260516 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c"] Oct 30 13:05:14.266591 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.266520 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-63f4e4-6d7899569f-k898c"] Oct 30 13:05:14.300643 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.300620 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8"] Oct 30 13:05:14.307316 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.307293 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-63f4e4-6f85b5b8b8-7k6r8"] Oct 30 13:05:14.314790 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.314771 4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-tmp-dir\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.314871 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.314792 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c6cr\" (UniqueName: \"kubernetes.io/projected/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-kube-api-access-5c6cr\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.314871 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.314803 4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d273864b-7bab-4471-8531-3855be5b6182-tls-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.314871 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.314814 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv5tq\" (UniqueName: \"kubernetes.io/projected/d273864b-7bab-4471-8531-3855be5b6182-kube-api-access-zv5tq\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.314871 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.314823 4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-model-cache\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.314871 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.314831 4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-dshm\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.314871 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.314838 4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-dshm\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.314871 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.314845 4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d273864b-7bab-4471-8531-3855be5b6182-home\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.314871 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.314853 4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1-tls-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.447009 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.446887 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" podUID="065b66a4-c80e-4dde-8bf6-4f14386e6d87" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:05:14.447009 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.446911 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" podUID="ab60f224-4948-4a2b-abf2-116cb1827625" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:05:14.501359 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.501333 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh"] Oct 30 13:05:14.513987 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.513959 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh"] Oct 30 13:05:14.551820 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.551792 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz"] Oct 30 13:05:14.555993 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.555969 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz"] Oct 30 13:05:14.973313 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:14.973271 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-2-w1-c1b1e8-kserve-54c95665f-g7zfz" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:15.002375 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:15.002344 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/premium-w1-c1b1e8-kserve-cc8778c7-b8rkh" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 13:05:15.090759 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:15.090728 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="416eb2ba-b5b4-4e25-a057-d870df27c1f7" path="/var/lib/kubelet/pods/416eb2ba-b5b4-4e25-a057-d870df27c1f7/volumes" Oct 30 13:05:15.091180 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:15.091164 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add4979c-c091-4dea-9853-615756907859" path="/var/lib/kubelet/pods/add4979c-c091-4dea-9853-615756907859/volumes" Oct 30 13:05:15.091671 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:15.091653 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d273864b-7bab-4471-8531-3855be5b6182" path="/var/lib/kubelet/pods/d273864b-7bab-4471-8531-3855be5b6182/volumes" Oct 30 13:05:15.092486 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:15.092465 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" path="/var/lib/kubelet/pods/ea95fef4-f1e3-4967-9c5f-b6133ae05eb1/volumes" Oct 30 13:05:16.676897 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:16.676861 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" podUID="42777387-98e9-4718-aa0e-dcdcb64b7a19" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:05:16.677288 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:16.676871 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" podUID="2dbcae46-3576-48d9-9691-50ddfaefd50e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:05:18.109244 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.109213 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" Oct 30 13:05:18.149899 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.149878 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" Oct 30 13:05:18.190554 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.190523 4468 generic.go:356] "Generic (PLEG): container finished" podID="ab60f224-4948-4a2b-abf2-116cb1827625" containerID="0a58c2de2b8acde39654c3f72789c720e25918f63388c024e6a5ba4eb9f19354" exitCode=0 Oct 30 13:05:18.190719 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.190579 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" Oct 30 13:05:18.190719 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.190606 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" event={"ID":"ab60f224-4948-4a2b-abf2-116cb1827625","Type":"ContainerDied","Data":"0a58c2de2b8acde39654c3f72789c720e25918f63388c024e6a5ba4eb9f19354"} Oct 30 13:05:18.190719 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.190657 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9" event={"ID":"ab60f224-4948-4a2b-abf2-116cb1827625","Type":"ContainerDied","Data":"1a29882f1cb29e77d78fe87df13b0d5b63dcb1eef7158b180ee91760f61d423d"} Oct 30 13:05:18.190719 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.190686 4468 scope.go:122] "RemoveContainer" containerID="0a58c2de2b8acde39654c3f72789c720e25918f63388c024e6a5ba4eb9f19354" Oct 30 13:05:18.192477 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.192453 4468 generic.go:356] "Generic (PLEG): container finished" podID="065b66a4-c80e-4dde-8bf6-4f14386e6d87" containerID="8da48b357841d8f1ea105eb6ccb75dafc58aa2b91555ebdea60ce07082755439" exitCode=0 Oct 30 13:05:18.192625 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.192502 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" Oct 30 13:05:18.192625 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.192548 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" event={"ID":"065b66a4-c80e-4dde-8bf6-4f14386e6d87","Type":"ContainerDied","Data":"8da48b357841d8f1ea105eb6ccb75dafc58aa2b91555ebdea60ce07082755439"} Oct 30 13:05:18.192625 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.192600 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg" event={"ID":"065b66a4-c80e-4dde-8bf6-4f14386e6d87","Type":"ContainerDied","Data":"827bca12bbb11531a1bf2d737cb746d50570047f070f5a73d252d1bdc37f24a8"} Oct 30 13:05:18.201875 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.201858 4468 scope.go:122] "RemoveContainer" containerID="0a58c2de2b8acde39654c3f72789c720e25918f63388c024e6a5ba4eb9f19354" Oct 30 13:05:18.202159 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:05:18.202138 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a58c2de2b8acde39654c3f72789c720e25918f63388c024e6a5ba4eb9f19354\": container with ID starting with 0a58c2de2b8acde39654c3f72789c720e25918f63388c024e6a5ba4eb9f19354 not found: ID does not exist" containerID="0a58c2de2b8acde39654c3f72789c720e25918f63388c024e6a5ba4eb9f19354" Oct 30 13:05:18.202234 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.202166 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a58c2de2b8acde39654c3f72789c720e25918f63388c024e6a5ba4eb9f19354"} err="failed to get container status \"0a58c2de2b8acde39654c3f72789c720e25918f63388c024e6a5ba4eb9f19354\": rpc error: code = NotFound desc = could not find container \"0a58c2de2b8acde39654c3f72789c720e25918f63388c024e6a5ba4eb9f19354\": container with ID starting with 0a58c2de2b8acde39654c3f72789c720e25918f63388c024e6a5ba4eb9f19354 not found: ID does not exist" Oct 30 13:05:18.202234 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.202182 4468 scope.go:122] "RemoveContainer" containerID="8da48b357841d8f1ea105eb6ccb75dafc58aa2b91555ebdea60ce07082755439" Oct 30 13:05:18.211699 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.211681 4468 scope.go:122] "RemoveContainer" containerID="8da48b357841d8f1ea105eb6ccb75dafc58aa2b91555ebdea60ce07082755439" Oct 30 13:05:18.212018 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:05:18.211998 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8da48b357841d8f1ea105eb6ccb75dafc58aa2b91555ebdea60ce07082755439\": container with ID starting with 8da48b357841d8f1ea105eb6ccb75dafc58aa2b91555ebdea60ce07082755439 not found: ID does not exist" containerID="8da48b357841d8f1ea105eb6ccb75dafc58aa2b91555ebdea60ce07082755439" Oct 30 13:05:18.212149 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.212038 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8da48b357841d8f1ea105eb6ccb75dafc58aa2b91555ebdea60ce07082755439"} err="failed to get container status \"8da48b357841d8f1ea105eb6ccb75dafc58aa2b91555ebdea60ce07082755439\": rpc error: code = NotFound desc = could not find container \"8da48b357841d8f1ea105eb6ccb75dafc58aa2b91555ebdea60ce07082755439\": container with ID starting with 8da48b357841d8f1ea105eb6ccb75dafc58aa2b91555ebdea60ce07082755439 not found: ID does not exist" Oct 30 13:05:18.260061 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.260034 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab60f224-4948-4a2b-abf2-116cb1827625-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab60f224-4948-4a2b-abf2-116cb1827625-plugins-config-volume\") pod \"ab60f224-4948-4a2b-abf2-116cb1827625\" (UID: \"ab60f224-4948-4a2b-abf2-116cb1827625\") " Oct 30 13:05:18.260247 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.260170 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab60f224-4948-4a2b-abf2-116cb1827625-kube-api-access-h4rnz\" (UniqueName: \"kubernetes.io/projected/ab60f224-4948-4a2b-abf2-116cb1827625-kube-api-access-h4rnz\") pod \"ab60f224-4948-4a2b-abf2-116cb1827625\" (UID: \"ab60f224-4948-4a2b-abf2-116cb1827625\") " Oct 30 13:05:18.260247 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.260236 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/065b66a4-c80e-4dde-8bf6-4f14386e6d87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065b66a4-c80e-4dde-8bf6-4f14386e6d87-plugins-config-volume\") pod \"065b66a4-c80e-4dde-8bf6-4f14386e6d87\" (UID: \"065b66a4-c80e-4dde-8bf6-4f14386e6d87\") " Oct 30 13:05:18.260366 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.260278 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/065b66a4-c80e-4dde-8bf6-4f14386e6d87-kube-api-access-5qd46\" (UniqueName: \"kubernetes.io/projected/065b66a4-c80e-4dde-8bf6-4f14386e6d87-kube-api-access-5qd46\") pod \"065b66a4-c80e-4dde-8bf6-4f14386e6d87\" (UID: \"065b66a4-c80e-4dde-8bf6-4f14386e6d87\") " Oct 30 13:05:18.260563 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.260532 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab60f224-4948-4a2b-abf2-116cb1827625-plugins-config-volume" pod "ab60f224-4948-4a2b-abf2-116cb1827625" (UID: "ab60f224-4948-4a2b-abf2-116cb1827625"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:18.260671 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.260568 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065b66a4-c80e-4dde-8bf6-4f14386e6d87-plugins-config-volume" pod "065b66a4-c80e-4dde-8bf6-4f14386e6d87" (UID: "065b66a4-c80e-4dde-8bf6-4f14386e6d87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:18.260933 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.260904 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab60f224-4948-4a2b-abf2-116cb1827625-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:18.261029 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.260937 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065b66a4-c80e-4dde-8bf6-4f14386e6d87-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:18.262524 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.262501 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab60f224-4948-4a2b-abf2-116cb1827625-kube-api-access-h4rnz" pod "ab60f224-4948-4a2b-abf2-116cb1827625" (UID: "ab60f224-4948-4a2b-abf2-116cb1827625"). InnerVolumeSpecName "kube-api-access-h4rnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:18.262617 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.262535 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065b66a4-c80e-4dde-8bf6-4f14386e6d87-kube-api-access-5qd46" pod "065b66a4-c80e-4dde-8bf6-4f14386e6d87" (UID: "065b66a4-c80e-4dde-8bf6-4f14386e6d87"). InnerVolumeSpecName "kube-api-access-5qd46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:18.362113 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.362047 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qd46\" (UniqueName: \"kubernetes.io/projected/065b66a4-c80e-4dde-8bf6-4f14386e6d87-kube-api-access-5qd46\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:18.362113 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.362104 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4rnz\" (UniqueName: \"kubernetes.io/projected/ab60f224-4948-4a2b-abf2-116cb1827625-kube-api-access-h4rnz\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:18.549324 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.549296 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg"] Oct 30 13:05:18.562215 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.562188 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0de4b9-f9547c6ff-fxsfg"] Oct 30 13:05:18.615258 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.615185 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9"] Oct 30 13:05:18.632478 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.632447 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0de4b9-546f7db65d-sgbl9"] Oct 30 13:05:18.783993 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:18.783960 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r"] Oct 30 13:05:19.094239 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:19.094205 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="065b66a4-c80e-4dde-8bf6-4f14386e6d87" path="/var/lib/kubelet/pods/065b66a4-c80e-4dde-8bf6-4f14386e6d87/volumes" Oct 30 13:05:19.094853 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:19.094836 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab60f224-4948-4a2b-abf2-116cb1827625" path="/var/lib/kubelet/pods/ab60f224-4948-4a2b-abf2-116cb1827625/volumes" Oct 30 13:05:23.607182 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:05:23.607059 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:05:23.784481 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:23.784419 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" podUID="49719ca9-b7db-4fd3-bcc8-feba390c6874" containerName="payload-processing" containerID="cri-o://03d0e084db11cfbf5acac856b70b115fb0c8f14b2335294caa22d1d54402acd6" gracePeriod=25 Oct 30 13:05:26.677288 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:26.677255 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" podUID="2dbcae46-3576-48d9-9691-50ddfaefd50e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:05:26.677693 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:26.677255 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" podUID="42777387-98e9-4718-aa0e-dcdcb64b7a19" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:05:29.669774 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:29.669748 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" Oct 30 13:05:29.694574 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:29.694555 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" Oct 30 13:05:29.814513 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:29.814483 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42777387-98e9-4718-aa0e-dcdcb64b7a19-kube-api-access-m4p42\" (UniqueName: \"kubernetes.io/projected/42777387-98e9-4718-aa0e-dcdcb64b7a19-kube-api-access-m4p42\") pod \"42777387-98e9-4718-aa0e-dcdcb64b7a19\" (UID: \"42777387-98e9-4718-aa0e-dcdcb64b7a19\") " Oct 30 13:05:29.814742 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:29.814543 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42777387-98e9-4718-aa0e-dcdcb64b7a19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42777387-98e9-4718-aa0e-dcdcb64b7a19-plugins-config-volume\") pod \"42777387-98e9-4718-aa0e-dcdcb64b7a19\" (UID: \"42777387-98e9-4718-aa0e-dcdcb64b7a19\") " Oct 30 13:05:29.814742 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:29.814580 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dbcae46-3576-48d9-9691-50ddfaefd50e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbcae46-3576-48d9-9691-50ddfaefd50e-plugins-config-volume\") pod \"2dbcae46-3576-48d9-9691-50ddfaefd50e\" (UID: \"2dbcae46-3576-48d9-9691-50ddfaefd50e\") " Oct 30 13:05:29.814742 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:29.814659 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dbcae46-3576-48d9-9691-50ddfaefd50e-kube-api-access-5qfpt\" (UniqueName: \"kubernetes.io/projected/2dbcae46-3576-48d9-9691-50ddfaefd50e-kube-api-access-5qfpt\") pod \"2dbcae46-3576-48d9-9691-50ddfaefd50e\" (UID: \"2dbcae46-3576-48d9-9691-50ddfaefd50e\") " Oct 30 13:05:29.815028 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:29.814994 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42777387-98e9-4718-aa0e-dcdcb64b7a19-plugins-config-volume" pod "42777387-98e9-4718-aa0e-dcdcb64b7a19" (UID: "42777387-98e9-4718-aa0e-dcdcb64b7a19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:29.815134 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:29.815047 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dbcae46-3576-48d9-9691-50ddfaefd50e-plugins-config-volume" pod "2dbcae46-3576-48d9-9691-50ddfaefd50e" (UID: "2dbcae46-3576-48d9-9691-50ddfaefd50e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:29.815244 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:29.815226 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42777387-98e9-4718-aa0e-dcdcb64b7a19-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:29.815295 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:29.815250 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbcae46-3576-48d9-9691-50ddfaefd50e-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:29.816759 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:29.816729 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42777387-98e9-4718-aa0e-dcdcb64b7a19-kube-api-access-m4p42" pod "42777387-98e9-4718-aa0e-dcdcb64b7a19" (UID: "42777387-98e9-4718-aa0e-dcdcb64b7a19"). InnerVolumeSpecName "kube-api-access-m4p42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:29.816865 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:29.816800 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dbcae46-3576-48d9-9691-50ddfaefd50e-kube-api-access-5qfpt" pod "2dbcae46-3576-48d9-9691-50ddfaefd50e" (UID: "2dbcae46-3576-48d9-9691-50ddfaefd50e"). InnerVolumeSpecName "kube-api-access-5qfpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:29.916522 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:29.916500 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4p42\" (UniqueName: \"kubernetes.io/projected/42777387-98e9-4718-aa0e-dcdcb64b7a19-kube-api-access-m4p42\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:29.916522 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:29.916524 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qfpt\" (UniqueName: \"kubernetes.io/projected/2dbcae46-3576-48d9-9691-50ddfaefd50e-kube-api-access-5qfpt\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:30.042161 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:30.042063 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g"] Oct 30 13:05:30.288989 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:30.288906 4468 generic.go:356] "Generic (PLEG): container finished" podID="2dbcae46-3576-48d9-9691-50ddfaefd50e" containerID="23e5ef21dca0e9e75d2ce2fe9ebd7d8406c3163ad3d33f32eab5d5904f7c04f0" exitCode=0 Oct 30 13:05:30.289247 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:30.289221 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" event={"ID":"2dbcae46-3576-48d9-9691-50ddfaefd50e","Type":"ContainerDied","Data":"23e5ef21dca0e9e75d2ce2fe9ebd7d8406c3163ad3d33f32eab5d5904f7c04f0"} Oct 30 13:05:30.289311 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:30.289288 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" event={"ID":"2dbcae46-3576-48d9-9691-50ddfaefd50e","Type":"ContainerDied","Data":"8efb43462b922332ca1a4124775fa42103b5ed32c40d8144511a4d9ac0a4bcd0"} Oct 30 13:05:30.289358 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:30.289320 4468 scope.go:122] "RemoveContainer" containerID="23e5ef21dca0e9e75d2ce2fe9ebd7d8406c3163ad3d33f32eab5d5904f7c04f0" Oct 30 13:05:30.289503 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:30.289487 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7" Oct 30 13:05:30.299244 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:30.299214 4468 generic.go:356] "Generic (PLEG): container finished" podID="42777387-98e9-4718-aa0e-dcdcb64b7a19" containerID="af1a9bc7dbe714cc2d93960e8d320b235695947cda49bae0e6b48c038d4b6401" exitCode=0 Oct 30 13:05:30.299420 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:30.299268 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" event={"ID":"42777387-98e9-4718-aa0e-dcdcb64b7a19","Type":"ContainerDied","Data":"af1a9bc7dbe714cc2d93960e8d320b235695947cda49bae0e6b48c038d4b6401"} Oct 30 13:05:30.299420 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:30.299314 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" event={"ID":"42777387-98e9-4718-aa0e-dcdcb64b7a19","Type":"ContainerDied","Data":"737e86857844149f7f9ab458fd8ac3bf4e6bc43df7f3a8dc38a371d92f4d2e4b"} Oct 30 13:05:30.299420 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:30.299392 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp" Oct 30 13:05:30.310510 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:30.310489 4468 scope.go:122] "RemoveContainer" containerID="23e5ef21dca0e9e75d2ce2fe9ebd7d8406c3163ad3d33f32eab5d5904f7c04f0" Oct 30 13:05:30.310813 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:05:30.310795 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23e5ef21dca0e9e75d2ce2fe9ebd7d8406c3163ad3d33f32eab5d5904f7c04f0\": container with ID starting with 23e5ef21dca0e9e75d2ce2fe9ebd7d8406c3163ad3d33f32eab5d5904f7c04f0 not found: ID does not exist" containerID="23e5ef21dca0e9e75d2ce2fe9ebd7d8406c3163ad3d33f32eab5d5904f7c04f0" Oct 30 13:05:30.310897 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:30.310819 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23e5ef21dca0e9e75d2ce2fe9ebd7d8406c3163ad3d33f32eab5d5904f7c04f0"} err="failed to get container status \"23e5ef21dca0e9e75d2ce2fe9ebd7d8406c3163ad3d33f32eab5d5904f7c04f0\": rpc error: code = NotFound desc = could not find container \"23e5ef21dca0e9e75d2ce2fe9ebd7d8406c3163ad3d33f32eab5d5904f7c04f0\": container with ID starting with 23e5ef21dca0e9e75d2ce2fe9ebd7d8406c3163ad3d33f32eab5d5904f7c04f0 not found: ID does not exist" Oct 30 13:05:30.310897 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:30.310835 4468 scope.go:122] "RemoveContainer" containerID="af1a9bc7dbe714cc2d93960e8d320b235695947cda49bae0e6b48c038d4b6401" Oct 30 13:05:30.320679 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:30.320657 4468 scope.go:122] "RemoveContainer" containerID="af1a9bc7dbe714cc2d93960e8d320b235695947cda49bae0e6b48c038d4b6401" Oct 30 13:05:30.320920 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:05:30.320901 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1a9bc7dbe714cc2d93960e8d320b235695947cda49bae0e6b48c038d4b6401\": container with ID starting with af1a9bc7dbe714cc2d93960e8d320b235695947cda49bae0e6b48c038d4b6401 not found: ID does not exist" containerID="af1a9bc7dbe714cc2d93960e8d320b235695947cda49bae0e6b48c038d4b6401" Oct 30 13:05:30.320986 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:30.320925 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1a9bc7dbe714cc2d93960e8d320b235695947cda49bae0e6b48c038d4b6401"} err="failed to get container status \"af1a9bc7dbe714cc2d93960e8d320b235695947cda49bae0e6b48c038d4b6401\": rpc error: code = NotFound desc = could not find container \"af1a9bc7dbe714cc2d93960e8d320b235695947cda49bae0e6b48c038d4b6401\": container with ID starting with af1a9bc7dbe714cc2d93960e8d320b235695947cda49bae0e6b48c038d4b6401 not found: ID does not exist" Oct 30 13:05:30.366457 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:30.366423 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp"] Oct 30 13:05:30.373393 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:30.373369 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c1b1e8-747b9c5d56-hf5pp"] Oct 30 13:05:30.405060 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:30.405037 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7"] Oct 30 13:05:30.409036 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:30.409014 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c1b1e8-54768d8dbc-fcmh7"] Oct 30 13:05:31.099672 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:31.099635 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dbcae46-3576-48d9-9691-50ddfaefd50e" path="/var/lib/kubelet/pods/2dbcae46-3576-48d9-9691-50ddfaefd50e/volumes" Oct 30 13:05:31.100375 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:31.100354 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42777387-98e9-4718-aa0e-dcdcb64b7a19" path="/var/lib/kubelet/pods/42777387-98e9-4718-aa0e-dcdcb64b7a19/volumes" Oct 30 13:05:33.035104 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:33.035043 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" podUID="49719ca9-b7db-4fd3-bcc8-feba390c6874" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:05:35.042363 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:35.042315 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" podUID="22fe1ef7-c64e-4523-be4c-f5460fadf274" containerName="payload-pre-processing" containerID="cri-o://52e87344ed7861777238f5ada60dcee04cf6355da16f5ccbe7e4293c5ef69dd6" gracePeriod=25 Oct 30 13:05:42.212242 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:42.212207 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" podUID="22fe1ef7-c64e-4523-be4c-f5460fadf274" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:05:43.035086 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:43.035043 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" podUID="49719ca9-b7db-4fd3-bcc8-feba390c6874" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:05:44.092704 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:44.092679 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" Oct 30 13:05:44.152981 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:44.150017 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49719ca9-b7db-4fd3-bcc8-feba390c6874-kube-api-access-6xxbn\" (UniqueName: \"kubernetes.io/projected/49719ca9-b7db-4fd3-bcc8-feba390c6874-kube-api-access-6xxbn\") pod \"49719ca9-b7db-4fd3-bcc8-feba390c6874\" (UID: \"49719ca9-b7db-4fd3-bcc8-feba390c6874\") " Oct 30 13:05:44.152981 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:44.150206 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49719ca9-b7db-4fd3-bcc8-feba390c6874-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49719ca9-b7db-4fd3-bcc8-feba390c6874-plugins-config-volume\") pod \"49719ca9-b7db-4fd3-bcc8-feba390c6874\" (UID: \"49719ca9-b7db-4fd3-bcc8-feba390c6874\") " Oct 30 13:05:44.152981 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:44.150615 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49719ca9-b7db-4fd3-bcc8-feba390c6874-plugins-config-volume" pod "49719ca9-b7db-4fd3-bcc8-feba390c6874" (UID: "49719ca9-b7db-4fd3-bcc8-feba390c6874"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:44.152981 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:44.150810 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49719ca9-b7db-4fd3-bcc8-feba390c6874-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:44.152981 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:44.152924 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49719ca9-b7db-4fd3-bcc8-feba390c6874-kube-api-access-6xxbn" pod "49719ca9-b7db-4fd3-bcc8-feba390c6874" (UID: "49719ca9-b7db-4fd3-bcc8-feba390c6874"). InnerVolumeSpecName "kube-api-access-6xxbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:44.251984 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:44.251955 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xxbn\" (UniqueName: \"kubernetes.io/projected/49719ca9-b7db-4fd3-bcc8-feba390c6874-kube-api-access-6xxbn\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:44.399603 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:44.399514 4468 generic.go:356] "Generic (PLEG): container finished" podID="49719ca9-b7db-4fd3-bcc8-feba390c6874" containerID="03d0e084db11cfbf5acac856b70b115fb0c8f14b2335294caa22d1d54402acd6" exitCode=0 Oct 30 13:05:44.399603 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:44.399559 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" Oct 30 13:05:44.399806 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:44.399593 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" event={"ID":"49719ca9-b7db-4fd3-bcc8-feba390c6874","Type":"ContainerDied","Data":"03d0e084db11cfbf5acac856b70b115fb0c8f14b2335294caa22d1d54402acd6"} Oct 30 13:05:44.399806 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:44.399650 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r" event={"ID":"49719ca9-b7db-4fd3-bcc8-feba390c6874","Type":"ContainerDied","Data":"ef4d9ecf7b437c591917d626ebd1245c67dfac72d71e3289acd2a079fc4a42df"} Oct 30 13:05:44.399806 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:44.399671 4468 scope.go:122] "RemoveContainer" containerID="03d0e084db11cfbf5acac856b70b115fb0c8f14b2335294caa22d1d54402acd6" Oct 30 13:05:44.410271 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:44.410253 4468 scope.go:122] "RemoveContainer" containerID="03d0e084db11cfbf5acac856b70b115fb0c8f14b2335294caa22d1d54402acd6" Oct 30 13:05:44.410522 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:05:44.410501 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d0e084db11cfbf5acac856b70b115fb0c8f14b2335294caa22d1d54402acd6\": container with ID starting with 03d0e084db11cfbf5acac856b70b115fb0c8f14b2335294caa22d1d54402acd6 not found: ID does not exist" containerID="03d0e084db11cfbf5acac856b70b115fb0c8f14b2335294caa22d1d54402acd6" Oct 30 13:05:44.410593 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:44.410535 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d0e084db11cfbf5acac856b70b115fb0c8f14b2335294caa22d1d54402acd6"} err="failed to get container status \"03d0e084db11cfbf5acac856b70b115fb0c8f14b2335294caa22d1d54402acd6\": rpc error: code = NotFound desc = could not find container \"03d0e084db11cfbf5acac856b70b115fb0c8f14b2335294caa22d1d54402acd6\": container with ID starting with 03d0e084db11cfbf5acac856b70b115fb0c8f14b2335294caa22d1d54402acd6 not found: ID does not exist" Oct 30 13:05:44.458131 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:44.458105 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r"] Oct 30 13:05:44.463737 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:44.463715 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-734ffe-578944ccfb-xk68r"] Oct 30 13:05:45.090042 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:45.090009 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49719ca9-b7db-4fd3-bcc8-feba390c6874" path="/var/lib/kubelet/pods/49719ca9-b7db-4fd3-bcc8-feba390c6874/volumes" Oct 30 13:05:50.340023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:50.339989 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4"] Oct 30 13:05:50.340591 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:50.340209 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" podUID="54a87f6f-513a-4a7b-a904-be6b4f218570" containerName="maas-api" containerID="cri-o://bfdb0b18005756cf21b5c931389fa31b272fd8f705027b5af65ab5117c9187a1" gracePeriod=30 Oct 30 13:05:50.611299 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:50.608626 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5"] Oct 30 13:05:50.644029 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:50.644004 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" Oct 30 13:05:50.730244 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:50.730204 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54a87f6f-513a-4a7b-a904-be6b4f218570-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54a87f6f-513a-4a7b-a904-be6b4f218570-openshift-service-ca\") pod \"54a87f6f-513a-4a7b-a904-be6b4f218570\" (UID: \"54a87f6f-513a-4a7b-a904-be6b4f218570\") " Oct 30 13:05:50.730420 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:50.730322 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54a87f6f-513a-4a7b-a904-be6b4f218570-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54a87f6f-513a-4a7b-a904-be6b4f218570-maas-api-tls\") pod \"54a87f6f-513a-4a7b-a904-be6b4f218570\" (UID: \"54a87f6f-513a-4a7b-a904-be6b4f218570\") " Oct 30 13:05:50.730420 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:50.730382 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a87f6f-513a-4a7b-a904-be6b4f218570-kube-api-access-cmhg9\" (UniqueName: \"kubernetes.io/projected/54a87f6f-513a-4a7b-a904-be6b4f218570-kube-api-access-cmhg9\") pod \"54a87f6f-513a-4a7b-a904-be6b4f218570\" (UID: \"54a87f6f-513a-4a7b-a904-be6b4f218570\") " Oct 30 13:05:50.730505 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:50.730420 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54a87f6f-513a-4a7b-a904-be6b4f218570-metrics-certs\" (UniqueName: \"kubernetes.io/secret/54a87f6f-513a-4a7b-a904-be6b4f218570-metrics-certs\") pod \"54a87f6f-513a-4a7b-a904-be6b4f218570\" (UID: \"54a87f6f-513a-4a7b-a904-be6b4f218570\") " Oct 30 13:05:50.732114 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:50.732055 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a87f6f-513a-4a7b-a904-be6b4f218570-openshift-service-ca" pod "54a87f6f-513a-4a7b-a904-be6b4f218570" (UID: "54a87f6f-513a-4a7b-a904-be6b4f218570"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:50.733708 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:50.733675 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a87f6f-513a-4a7b-a904-be6b4f218570-metrics-certs" pod "54a87f6f-513a-4a7b-a904-be6b4f218570" (UID: "54a87f6f-513a-4a7b-a904-be6b4f218570"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:50.733970 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:50.733942 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a87f6f-513a-4a7b-a904-be6b4f218570-kube-api-access-cmhg9" pod "54a87f6f-513a-4a7b-a904-be6b4f218570" (UID: "54a87f6f-513a-4a7b-a904-be6b4f218570"). InnerVolumeSpecName "kube-api-access-cmhg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:50.734366 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:50.734346 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a87f6f-513a-4a7b-a904-be6b4f218570-maas-api-tls" pod "54a87f6f-513a-4a7b-a904-be6b4f218570" (UID: "54a87f6f-513a-4a7b-a904-be6b4f218570"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:50.871644 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:50.871404 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54a87f6f-513a-4a7b-a904-be6b4f218570-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:50.871644 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:50.871471 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54a87f6f-513a-4a7b-a904-be6b4f218570-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:50.871644 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:50.871491 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmhg9\" (UniqueName: \"kubernetes.io/projected/54a87f6f-513a-4a7b-a904-be6b4f218570-kube-api-access-cmhg9\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:50.871644 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:50.871506 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54a87f6f-513a-4a7b-a904-be6b4f218570-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:51.448164 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:51.448129 4468 generic.go:356] "Generic (PLEG): container finished" podID="54a87f6f-513a-4a7b-a904-be6b4f218570" containerID="bfdb0b18005756cf21b5c931389fa31b272fd8f705027b5af65ab5117c9187a1" exitCode=0 Oct 30 13:05:51.448614 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:51.448184 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" Oct 30 13:05:51.448614 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:51.448222 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" event={"ID":"54a87f6f-513a-4a7b-a904-be6b4f218570","Type":"ContainerDied","Data":"bfdb0b18005756cf21b5c931389fa31b272fd8f705027b5af65ab5117c9187a1"} Oct 30 13:05:51.448614 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:51.448285 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4" event={"ID":"54a87f6f-513a-4a7b-a904-be6b4f218570","Type":"ContainerDied","Data":"553b3497fc4e8507d4dccbfa14b3e76f5c8d24d10e9f80cebc4eb0c0df25c024"} Oct 30 13:05:51.448614 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:51.448316 4468 scope.go:122] "RemoveContainer" containerID="bfdb0b18005756cf21b5c931389fa31b272fd8f705027b5af65ab5117c9187a1" Oct 30 13:05:51.459330 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:51.459297 4468 scope.go:122] "RemoveContainer" containerID="bfdb0b18005756cf21b5c931389fa31b272fd8f705027b5af65ab5117c9187a1" Oct 30 13:05:51.459625 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:05:51.459601 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfdb0b18005756cf21b5c931389fa31b272fd8f705027b5af65ab5117c9187a1\": container with ID starting with bfdb0b18005756cf21b5c931389fa31b272fd8f705027b5af65ab5117c9187a1 not found: ID does not exist" containerID="bfdb0b18005756cf21b5c931389fa31b272fd8f705027b5af65ab5117c9187a1" Oct 30 13:05:51.459697 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:51.459633 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfdb0b18005756cf21b5c931389fa31b272fd8f705027b5af65ab5117c9187a1"} err="failed to get container status \"bfdb0b18005756cf21b5c931389fa31b272fd8f705027b5af65ab5117c9187a1\": rpc error: code = NotFound desc = could not find container \"bfdb0b18005756cf21b5c931389fa31b272fd8f705027b5af65ab5117c9187a1\": container with ID starting with bfdb0b18005756cf21b5c931389fa31b272fd8f705027b5af65ab5117c9187a1 not found: ID does not exist" Oct 30 13:05:51.507047 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:51.507018 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4"] Oct 30 13:05:51.521060 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:51.521035 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-734ffe-5b8fbf9bc5-sspg4"] Oct 30 13:05:52.212114 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:52.212058 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" podUID="22fe1ef7-c64e-4523-be4c-f5460fadf274" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:05:52.394934 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:52.394899 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5"] Oct 30 13:05:52.429114 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:52.429084 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs"] Oct 30 13:05:53.092686 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:53.092639 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54a87f6f-513a-4a7b-a904-be6b4f218570" path="/var/lib/kubelet/pods/54a87f6f-513a-4a7b-a904-be6b4f218570/volumes" Oct 30 13:05:53.598110 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:05:53.598062 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:05:55.332853 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.332828 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" Oct 30 13:05:55.456280 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.456244 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22fe1ef7-c64e-4523-be4c-f5460fadf274-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fe1ef7-c64e-4523-be4c-f5460fadf274-plugins-config-volume\") pod \"22fe1ef7-c64e-4523-be4c-f5460fadf274\" (UID: \"22fe1ef7-c64e-4523-be4c-f5460fadf274\") " Oct 30 13:05:55.456494 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.456301 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fe1ef7-c64e-4523-be4c-f5460fadf274-kube-api-access-vt6pl\" (UniqueName: \"kubernetes.io/projected/22fe1ef7-c64e-4523-be4c-f5460fadf274-kube-api-access-vt6pl\") pod \"22fe1ef7-c64e-4523-be4c-f5460fadf274\" (UID: \"22fe1ef7-c64e-4523-be4c-f5460fadf274\") " Oct 30 13:05:55.456640 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.456613 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fe1ef7-c64e-4523-be4c-f5460fadf274-plugins-config-volume" pod "22fe1ef7-c64e-4523-be4c-f5460fadf274" (UID: "22fe1ef7-c64e-4523-be4c-f5460fadf274"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:55.456816 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.456801 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fe1ef7-c64e-4523-be4c-f5460fadf274-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:55.458548 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.458515 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fe1ef7-c64e-4523-be4c-f5460fadf274-kube-api-access-vt6pl" pod "22fe1ef7-c64e-4523-be4c-f5460fadf274" (UID: "22fe1ef7-c64e-4523-be4c-f5460fadf274"). InnerVolumeSpecName "kube-api-access-vt6pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:55.478682 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.478656 4468 generic.go:356] "Generic (PLEG): container finished" podID="22fe1ef7-c64e-4523-be4c-f5460fadf274" containerID="52e87344ed7861777238f5ada60dcee04cf6355da16f5ccbe7e4293c5ef69dd6" exitCode=0 Oct 30 13:05:55.478800 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.478694 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" event={"ID":"22fe1ef7-c64e-4523-be4c-f5460fadf274","Type":"ContainerDied","Data":"52e87344ed7861777238f5ada60dcee04cf6355da16f5ccbe7e4293c5ef69dd6"} Oct 30 13:05:55.478800 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.478705 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" Oct 30 13:05:55.478800 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.478735 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g" event={"ID":"22fe1ef7-c64e-4523-be4c-f5460fadf274","Type":"ContainerDied","Data":"aab019b204deee219bd861987658d9a5002b60212698fb498d9555459a963cb2"} Oct 30 13:05:55.478800 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.478757 4468 scope.go:122] "RemoveContainer" containerID="52e87344ed7861777238f5ada60dcee04cf6355da16f5ccbe7e4293c5ef69dd6" Oct 30 13:05:55.489303 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.489285 4468 scope.go:122] "RemoveContainer" containerID="52e87344ed7861777238f5ada60dcee04cf6355da16f5ccbe7e4293c5ef69dd6" Oct 30 13:05:55.489599 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:05:55.489579 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52e87344ed7861777238f5ada60dcee04cf6355da16f5ccbe7e4293c5ef69dd6\": container with ID starting with 52e87344ed7861777238f5ada60dcee04cf6355da16f5ccbe7e4293c5ef69dd6 not found: ID does not exist" containerID="52e87344ed7861777238f5ada60dcee04cf6355da16f5ccbe7e4293c5ef69dd6" Oct 30 13:05:55.489657 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.489604 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52e87344ed7861777238f5ada60dcee04cf6355da16f5ccbe7e4293c5ef69dd6"} err="failed to get container status \"52e87344ed7861777238f5ada60dcee04cf6355da16f5ccbe7e4293c5ef69dd6\": rpc error: code = NotFound desc = could not find container \"52e87344ed7861777238f5ada60dcee04cf6355da16f5ccbe7e4293c5ef69dd6\": container with ID starting with 52e87344ed7861777238f5ada60dcee04cf6355da16f5ccbe7e4293c5ef69dd6 not found: ID does not exist" Oct 30 13:05:55.537340 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.537295 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g"] Oct 30 13:05:55.541665 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.541641 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-befaa6-86b4b8bcf9-jd94g"] Oct 30 13:05:55.557868 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.557818 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt6pl\" (UniqueName: \"kubernetes.io/projected/22fe1ef7-c64e-4523-be4c-f5460fadf274-kube-api-access-vt6pl\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:55.609320 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.609284 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5" podUID="6bac85e8-7e06-4933-a757-4739e81a0afd" containerName="payload-pre-processing" containerID="cri-o://40007ecdeeb66519502bf7ec525b445c6cc72f317346485fbceab5312d4d707a" gracePeriod=25 Oct 30 13:05:55.856184 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.856154 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5" Oct 30 13:05:55.963568 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.963538 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bac85e8-7e06-4933-a757-4739e81a0afd-kube-api-access-4t6vh\" (UniqueName: \"kubernetes.io/projected/6bac85e8-7e06-4933-a757-4739e81a0afd-kube-api-access-4t6vh\") pod \"6bac85e8-7e06-4933-a757-4739e81a0afd\" (UID: \"6bac85e8-7e06-4933-a757-4739e81a0afd\") " Oct 30 13:05:55.963772 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.963623 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bac85e8-7e06-4933-a757-4739e81a0afd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bac85e8-7e06-4933-a757-4739e81a0afd-plugins-config-volume\") pod \"6bac85e8-7e06-4933-a757-4739e81a0afd\" (UID: \"6bac85e8-7e06-4933-a757-4739e81a0afd\") " Oct 30 13:05:55.964027 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.964003 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bac85e8-7e06-4933-a757-4739e81a0afd-plugins-config-volume" pod "6bac85e8-7e06-4933-a757-4739e81a0afd" (UID: "6bac85e8-7e06-4933-a757-4739e81a0afd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:55.965695 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:55.965675 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bac85e8-7e06-4933-a757-4739e81a0afd-kube-api-access-4t6vh" pod "6bac85e8-7e06-4933-a757-4739e81a0afd" (UID: "6bac85e8-7e06-4933-a757-4739e81a0afd"). InnerVolumeSpecName "kube-api-access-4t6vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:56.065759 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:56.065724 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bac85e8-7e06-4933-a757-4739e81a0afd-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:56.065876 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:56.065764 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t6vh\" (UniqueName: \"kubernetes.io/projected/6bac85e8-7e06-4933-a757-4739e81a0afd-kube-api-access-4t6vh\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:56.487428 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:56.487390 4468 generic.go:356] "Generic (PLEG): container finished" podID="6bac85e8-7e06-4933-a757-4739e81a0afd" containerID="40007ecdeeb66519502bf7ec525b445c6cc72f317346485fbceab5312d4d707a" exitCode=0 Oct 30 13:05:56.487904 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:56.487454 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5" Oct 30 13:05:56.487904 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:56.487479 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5" event={"ID":"6bac85e8-7e06-4933-a757-4739e81a0afd","Type":"ContainerDied","Data":"40007ecdeeb66519502bf7ec525b445c6cc72f317346485fbceab5312d4d707a"} Oct 30 13:05:56.487904 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:56.487550 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5" event={"ID":"6bac85e8-7e06-4933-a757-4739e81a0afd","Type":"ContainerDied","Data":"1482e5bff9b04d34f93f992e2386922101c518dae0edbe2a181c78492c0e21f7"} Oct 30 13:05:56.487904 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:56.487580 4468 scope.go:122] "RemoveContainer" containerID="40007ecdeeb66519502bf7ec525b445c6cc72f317346485fbceab5312d4d707a" Oct 30 13:05:56.502672 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:56.502652 4468 scope.go:122] "RemoveContainer" containerID="40007ecdeeb66519502bf7ec525b445c6cc72f317346485fbceab5312d4d707a" Oct 30 13:05:56.502969 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:05:56.502948 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40007ecdeeb66519502bf7ec525b445c6cc72f317346485fbceab5312d4d707a\": container with ID starting with 40007ecdeeb66519502bf7ec525b445c6cc72f317346485fbceab5312d4d707a not found: ID does not exist" containerID="40007ecdeeb66519502bf7ec525b445c6cc72f317346485fbceab5312d4d707a" Oct 30 13:05:56.503057 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:56.502976 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40007ecdeeb66519502bf7ec525b445c6cc72f317346485fbceab5312d4d707a"} err="failed to get container status \"40007ecdeeb66519502bf7ec525b445c6cc72f317346485fbceab5312d4d707a\": rpc error: code = NotFound desc = could not find container \"40007ecdeeb66519502bf7ec525b445c6cc72f317346485fbceab5312d4d707a\": container with ID starting with 40007ecdeeb66519502bf7ec525b445c6cc72f317346485fbceab5312d4d707a not found: ID does not exist" Oct 30 13:05:56.555093 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:56.555051 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5"] Oct 30 13:05:56.567991 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:56.567967 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-73bafc-5754b675dc-728g5"] Oct 30 13:05:57.090241 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:57.090203 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22fe1ef7-c64e-4523-be4c-f5460fadf274" path="/var/lib/kubelet/pods/22fe1ef7-c64e-4523-be4c-f5460fadf274/volumes" Oct 30 13:05:57.090645 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:57.090628 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bac85e8-7e06-4933-a757-4739e81a0afd" path="/var/lib/kubelet/pods/6bac85e8-7e06-4933-a757-4739e81a0afd/volumes" Oct 30 13:05:57.395727 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:57.395646 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5" podUID="cd5da4da-cc71-4a46-a697-396a96373776" containerName="payload-processing" containerID="cri-o://bf19e83d66b3710474592acb381359524013eb6a71b92b8df05f2a7ab754fa6e" gracePeriod=25 Oct 30 13:05:57.430285 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:57.430253 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" podUID="6d9cb487-201b-4bc1-ba55-c3e7c4306174" containerName="payload-pre-processing" containerID="cri-o://7aa04a4d8bc3dfb0991a5bd27a2612dec6397e28f04ca4ffe0cf42f168c0739d" gracePeriod=25 Oct 30 13:05:57.678621 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:57.678593 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5" Oct 30 13:05:57.786989 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:57.786952 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd5da4da-cc71-4a46-a697-396a96373776-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd5da4da-cc71-4a46-a697-396a96373776-plugins-config-volume\") pod \"cd5da4da-cc71-4a46-a697-396a96373776\" (UID: \"cd5da4da-cc71-4a46-a697-396a96373776\") " Oct 30 13:05:57.787162 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:57.787112 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd5da4da-cc71-4a46-a697-396a96373776-kube-api-access-wtx4m\" (UniqueName: \"kubernetes.io/projected/cd5da4da-cc71-4a46-a697-396a96373776-kube-api-access-wtx4m\") pod \"cd5da4da-cc71-4a46-a697-396a96373776\" (UID: \"cd5da4da-cc71-4a46-a697-396a96373776\") " Oct 30 13:05:57.787445 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:57.787420 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd5da4da-cc71-4a46-a697-396a96373776-plugins-config-volume" pod "cd5da4da-cc71-4a46-a697-396a96373776" (UID: "cd5da4da-cc71-4a46-a697-396a96373776"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:57.789149 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:57.789131 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd5da4da-cc71-4a46-a697-396a96373776-kube-api-access-wtx4m" pod "cd5da4da-cc71-4a46-a697-396a96373776" (UID: "cd5da4da-cc71-4a46-a697-396a96373776"). InnerVolumeSpecName "kube-api-access-wtx4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:57.888893 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:57.888862 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtx4m\" (UniqueName: \"kubernetes.io/projected/cd5da4da-cc71-4a46-a697-396a96373776-kube-api-access-wtx4m\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:57.888893 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:57.888889 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd5da4da-cc71-4a46-a697-396a96373776-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:05:58.505510 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:58.505473 4468 generic.go:356] "Generic (PLEG): container finished" podID="cd5da4da-cc71-4a46-a697-396a96373776" containerID="bf19e83d66b3710474592acb381359524013eb6a71b92b8df05f2a7ab754fa6e" exitCode=0 Oct 30 13:05:58.505706 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:58.505535 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5" Oct 30 13:05:58.505706 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:58.505568 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5" event={"ID":"cd5da4da-cc71-4a46-a697-396a96373776","Type":"ContainerDied","Data":"bf19e83d66b3710474592acb381359524013eb6a71b92b8df05f2a7ab754fa6e"} Oct 30 13:05:58.505706 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:58.505638 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5" event={"ID":"cd5da4da-cc71-4a46-a697-396a96373776","Type":"ContainerDied","Data":"e0048206b749574c16cc813b6c13980c7529ce8b98ea114069d35a4b871da1fc"} Oct 30 13:05:58.505706 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:58.505652 4468 scope.go:122] "RemoveContainer" containerID="bf19e83d66b3710474592acb381359524013eb6a71b92b8df05f2a7ab754fa6e" Oct 30 13:05:58.518248 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:58.518102 4468 scope.go:122] "RemoveContainer" containerID="bf19e83d66b3710474592acb381359524013eb6a71b92b8df05f2a7ab754fa6e" Oct 30 13:05:58.518420 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:05:58.518398 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf19e83d66b3710474592acb381359524013eb6a71b92b8df05f2a7ab754fa6e\": container with ID starting with bf19e83d66b3710474592acb381359524013eb6a71b92b8df05f2a7ab754fa6e not found: ID does not exist" containerID="bf19e83d66b3710474592acb381359524013eb6a71b92b8df05f2a7ab754fa6e" Oct 30 13:05:58.518472 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:58.518430 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf19e83d66b3710474592acb381359524013eb6a71b92b8df05f2a7ab754fa6e"} err="failed to get container status \"bf19e83d66b3710474592acb381359524013eb6a71b92b8df05f2a7ab754fa6e\": rpc error: code = NotFound desc = could not find container \"bf19e83d66b3710474592acb381359524013eb6a71b92b8df05f2a7ab754fa6e\": container with ID starting with bf19e83d66b3710474592acb381359524013eb6a71b92b8df05f2a7ab754fa6e not found: ID does not exist" Oct 30 13:05:58.563548 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:58.563517 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5"] Oct 30 13:05:58.576236 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:58.576214 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1333c8-68778777f4-2kxd5"] Oct 30 13:05:59.090649 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:05:59.090617 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd5da4da-cc71-4a46-a697-396a96373776" path="/var/lib/kubelet/pods/cd5da4da-cc71-4a46-a697-396a96373776/volumes" Oct 30 13:06:06.283155 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:06.283117 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" podUID="6d9cb487-201b-4bc1-ba55-c3e7c4306174" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:06:07.428576 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:07.428539 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79"] Oct 30 13:06:07.429015 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:07.428795 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" podUID="bcddfa21-70dc-407b-803f-c88f7ae0d27e" containerName="maas-api" containerID="cri-o://5129c6cf22d0f5d84be956fe26cc114a6dd9e009ea12898c3d6297134a30d0f4" gracePeriod=30 Oct 30 13:06:07.593912 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:07.593882 4468 generic.go:356] "Generic (PLEG): container finished" podID="bcddfa21-70dc-407b-803f-c88f7ae0d27e" containerID="5129c6cf22d0f5d84be956fe26cc114a6dd9e009ea12898c3d6297134a30d0f4" exitCode=0 Oct 30 13:06:07.594055 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:07.593963 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" event={"ID":"bcddfa21-70dc-407b-803f-c88f7ae0d27e","Type":"ContainerDied","Data":"5129c6cf22d0f5d84be956fe26cc114a6dd9e009ea12898c3d6297134a30d0f4"} Oct 30 13:06:07.721566 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:07.721543 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" Oct 30 13:06:07.821451 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:07.821422 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcddfa21-70dc-407b-803f-c88f7ae0d27e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcddfa21-70dc-407b-803f-c88f7ae0d27e-maas-api-tls\") pod \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\" (UID: \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\") " Oct 30 13:06:07.821451 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:07.821457 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcddfa21-70dc-407b-803f-c88f7ae0d27e-kube-api-access-g7skz\" (UniqueName: \"kubernetes.io/projected/bcddfa21-70dc-407b-803f-c88f7ae0d27e-kube-api-access-g7skz\") pod \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\" (UID: \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\") " Oct 30 13:06:07.821674 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:07.821518 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcddfa21-70dc-407b-803f-c88f7ae0d27e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcddfa21-70dc-407b-803f-c88f7ae0d27e-metrics-certs\") pod \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\" (UID: \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\") " Oct 30 13:06:07.821674 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:07.821546 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcddfa21-70dc-407b-803f-c88f7ae0d27e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcddfa21-70dc-407b-803f-c88f7ae0d27e-openshift-service-ca\") pod \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\" (UID: \"bcddfa21-70dc-407b-803f-c88f7ae0d27e\") " Oct 30 13:06:07.822017 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:07.821987 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcddfa21-70dc-407b-803f-c88f7ae0d27e-openshift-service-ca" pod "bcddfa21-70dc-407b-803f-c88f7ae0d27e" (UID: "bcddfa21-70dc-407b-803f-c88f7ae0d27e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:07.823760 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:07.823737 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcddfa21-70dc-407b-803f-c88f7ae0d27e-kube-api-access-g7skz" pod "bcddfa21-70dc-407b-803f-c88f7ae0d27e" (UID: "bcddfa21-70dc-407b-803f-c88f7ae0d27e"). InnerVolumeSpecName "kube-api-access-g7skz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:07.824063 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:07.824031 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcddfa21-70dc-407b-803f-c88f7ae0d27e-maas-api-tls" pod "bcddfa21-70dc-407b-803f-c88f7ae0d27e" (UID: "bcddfa21-70dc-407b-803f-c88f7ae0d27e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:07.824063 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:07.824046 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcddfa21-70dc-407b-803f-c88f7ae0d27e-metrics-certs" pod "bcddfa21-70dc-407b-803f-c88f7ae0d27e" (UID: "bcddfa21-70dc-407b-803f-c88f7ae0d27e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:07.923317 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:07.923279 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcddfa21-70dc-407b-803f-c88f7ae0d27e-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:07.923317 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:07.923315 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcddfa21-70dc-407b-803f-c88f7ae0d27e-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:07.923317 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:07.923326 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcddfa21-70dc-407b-803f-c88f7ae0d27e-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:07.923563 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:07.923334 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7skz\" (UniqueName: \"kubernetes.io/projected/bcddfa21-70dc-407b-803f-c88f7ae0d27e-kube-api-access-g7skz\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:08.601944 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:08.601916 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" Oct 30 13:06:08.602415 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:08.601910 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79" event={"ID":"bcddfa21-70dc-407b-803f-c88f7ae0d27e","Type":"ContainerDied","Data":"43e6dde9a1a0c198e9a2887138768c50a072cd0c2f2beb37856d8eba45d503f4"} Oct 30 13:06:08.602415 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:08.602065 4468 scope.go:122] "RemoveContainer" containerID="5129c6cf22d0f5d84be956fe26cc114a6dd9e009ea12898c3d6297134a30d0f4" Oct 30 13:06:08.660975 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:08.660946 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79"] Oct 30 13:06:08.674559 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:08.674539 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c1b1e8-7d6779f549-btl79"] Oct 30 13:06:09.091406 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:09.091375 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcddfa21-70dc-407b-803f-c88f7ae0d27e" path="/var/lib/kubelet/pods/bcddfa21-70dc-407b-803f-c88f7ae0d27e/volumes" Oct 30 13:06:09.652456 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:09.652415 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v"] Oct 30 13:06:09.652948 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:09.652681 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" podUID="557e9c97-61e1-4358-8249-c79d946c5a9c" containerName="maas-api" containerID="cri-o://6f989341796d1760cb1bef6c31ccf07cb626d2ccd509224c25160d203403752c" gracePeriod=30 Oct 30 13:06:09.921135 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:09.917656 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs"] Oct 30 13:06:09.936187 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:09.936158 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" Oct 30 13:06:09.984166 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:09.984133 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2"] Oct 30 13:06:10.048384 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.048343 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/557e9c97-61e1-4358-8249-c79d946c5a9c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/557e9c97-61e1-4358-8249-c79d946c5a9c-openshift-service-ca\") pod \"557e9c97-61e1-4358-8249-c79d946c5a9c\" (UID: \"557e9c97-61e1-4358-8249-c79d946c5a9c\") " Oct 30 13:06:10.048583 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.048502 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/557e9c97-61e1-4358-8249-c79d946c5a9c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/557e9c97-61e1-4358-8249-c79d946c5a9c-maas-api-tls\") pod \"557e9c97-61e1-4358-8249-c79d946c5a9c\" (UID: \"557e9c97-61e1-4358-8249-c79d946c5a9c\") " Oct 30 13:06:10.048583 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.048548 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/557e9c97-61e1-4358-8249-c79d946c5a9c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/557e9c97-61e1-4358-8249-c79d946c5a9c-metrics-certs\") pod \"557e9c97-61e1-4358-8249-c79d946c5a9c\" (UID: \"557e9c97-61e1-4358-8249-c79d946c5a9c\") " Oct 30 13:06:10.048713 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.048616 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/557e9c97-61e1-4358-8249-c79d946c5a9c-kube-api-access-48q7t\" (UniqueName: \"kubernetes.io/projected/557e9c97-61e1-4358-8249-c79d946c5a9c-kube-api-access-48q7t\") pod \"557e9c97-61e1-4358-8249-c79d946c5a9c\" (UID: \"557e9c97-61e1-4358-8249-c79d946c5a9c\") " Oct 30 13:06:10.048774 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.048739 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/557e9c97-61e1-4358-8249-c79d946c5a9c-openshift-service-ca" pod "557e9c97-61e1-4358-8249-c79d946c5a9c" (UID: "557e9c97-61e1-4358-8249-c79d946c5a9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:10.049192 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.049167 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/557e9c97-61e1-4358-8249-c79d946c5a9c-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:10.051045 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.051018 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/557e9c97-61e1-4358-8249-c79d946c5a9c-metrics-certs" pod "557e9c97-61e1-4358-8249-c79d946c5a9c" (UID: "557e9c97-61e1-4358-8249-c79d946c5a9c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:10.051232 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.051204 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/557e9c97-61e1-4358-8249-c79d946c5a9c-kube-api-access-48q7t" pod "557e9c97-61e1-4358-8249-c79d946c5a9c" (UID: "557e9c97-61e1-4358-8249-c79d946c5a9c"). InnerVolumeSpecName "kube-api-access-48q7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:10.051340 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.051272 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/557e9c97-61e1-4358-8249-c79d946c5a9c-maas-api-tls" pod "557e9c97-61e1-4358-8249-c79d946c5a9c" (UID: "557e9c97-61e1-4358-8249-c79d946c5a9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:10.151274 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.151245 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48q7t\" (UniqueName: \"kubernetes.io/projected/557e9c97-61e1-4358-8249-c79d946c5a9c-kube-api-access-48q7t\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:10.151274 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.151270 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/557e9c97-61e1-4358-8249-c79d946c5a9c-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:10.151274 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.151279 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/557e9c97-61e1-4358-8249-c79d946c5a9c-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:10.618704 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.618667 4468 generic.go:356] "Generic (PLEG): container finished" podID="557e9c97-61e1-4358-8249-c79d946c5a9c" containerID="6f989341796d1760cb1bef6c31ccf07cb626d2ccd509224c25160d203403752c" exitCode=0 Oct 30 13:06:10.618892 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.618740 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" Oct 30 13:06:10.618892 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.618758 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" event={"ID":"557e9c97-61e1-4358-8249-c79d946c5a9c","Type":"ContainerDied","Data":"6f989341796d1760cb1bef6c31ccf07cb626d2ccd509224c25160d203403752c"} Oct 30 13:06:10.618892 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.618814 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v" event={"ID":"557e9c97-61e1-4358-8249-c79d946c5a9c","Type":"ContainerDied","Data":"129a4ee6321885d684a6b7eb49f40b66f337b27d7f6f5c43361990404aea7a29"} Oct 30 13:06:10.618892 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.618839 4468 scope.go:122] "RemoveContainer" containerID="6f989341796d1760cb1bef6c31ccf07cb626d2ccd509224c25160d203403752c" Oct 30 13:06:10.630474 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.630458 4468 scope.go:122] "RemoveContainer" containerID="6f989341796d1760cb1bef6c31ccf07cb626d2ccd509224c25160d203403752c" Oct 30 13:06:10.630745 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:06:10.630723 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f989341796d1760cb1bef6c31ccf07cb626d2ccd509224c25160d203403752c\": container with ID starting with 6f989341796d1760cb1bef6c31ccf07cb626d2ccd509224c25160d203403752c not found: ID does not exist" containerID="6f989341796d1760cb1bef6c31ccf07cb626d2ccd509224c25160d203403752c" Oct 30 13:06:10.630852 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.630752 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f989341796d1760cb1bef6c31ccf07cb626d2ccd509224c25160d203403752c"} err="failed to get container status \"6f989341796d1760cb1bef6c31ccf07cb626d2ccd509224c25160d203403752c\": rpc error: code = NotFound desc = could not find container \"6f989341796d1760cb1bef6c31ccf07cb626d2ccd509224c25160d203403752c\": container with ID starting with 6f989341796d1760cb1bef6c31ccf07cb626d2ccd509224c25160d203403752c not found: ID does not exist" Oct 30 13:06:10.676775 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.676748 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v"] Oct 30 13:06:10.683494 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:10.683472 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-73bafc-58ddb75b8f-dtr4v"] Oct 30 13:06:11.091173 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:11.091142 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="557e9c97-61e1-4358-8249-c79d946c5a9c" path="/var/lib/kubelet/pods/557e9c97-61e1-4358-8249-c79d946c5a9c/volumes" Oct 30 13:06:13.419078 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:13.419047 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d"] Oct 30 13:06:13.439544 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:13.439514 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22"] Oct 30 13:06:14.919036 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:14.918997 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" podUID="bc9a08f6-1257-4e45-9a7d-9687d6554377" containerName="payload-processing" containerID="cri-o://26c1b443cb3f7f1effe37bd0a04f9878fb15d08bfa54d811287c7dd53f089a87" gracePeriod=25 Oct 30 13:06:14.984613 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:14.984577 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" podUID="5db888ab-7750-41f1-9a4c-3add0f15cc6d" containerName="payload-pre-processing" containerID="cri-o://0dabc8ba218f01b32a71860b65add6d6de98fcb2ed635627e028295738d7e7b5" gracePeriod=25 Oct 30 13:06:15.341858 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.341777 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r"] Oct 30 13:06:15.342408 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.342369 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" podUID="7c7c9b69-2356-49ba-ad43-2a8773f48a46" containerName="maas-api" containerID="cri-o://dbbb2d19c66178b0c82659b2db6f18de3fabeb760dd451646705ad61d54e27e2" gracePeriod=30 Oct 30 13:06:15.619227 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.619204 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" Oct 30 13:06:15.672741 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.672713 4468 generic.go:356] "Generic (PLEG): container finished" podID="7c7c9b69-2356-49ba-ad43-2a8773f48a46" containerID="dbbb2d19c66178b0c82659b2db6f18de3fabeb760dd451646705ad61d54e27e2" exitCode=0 Oct 30 13:06:15.672910 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.672767 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" Oct 30 13:06:15.672910 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.672796 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" event={"ID":"7c7c9b69-2356-49ba-ad43-2a8773f48a46","Type":"ContainerDied","Data":"dbbb2d19c66178b0c82659b2db6f18de3fabeb760dd451646705ad61d54e27e2"} Oct 30 13:06:15.672910 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.672870 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r" event={"ID":"7c7c9b69-2356-49ba-ad43-2a8773f48a46","Type":"ContainerDied","Data":"0d2389018f1b72476889be216160a2edadf1eb020b68371b75d2862826eaeadf"} Oct 30 13:06:15.672910 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.672899 4468 scope.go:122] "RemoveContainer" containerID="dbbb2d19c66178b0c82659b2db6f18de3fabeb760dd451646705ad61d54e27e2" Oct 30 13:06:15.683158 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.683137 4468 scope.go:122] "RemoveContainer" containerID="dbbb2d19c66178b0c82659b2db6f18de3fabeb760dd451646705ad61d54e27e2" Oct 30 13:06:15.683458 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:06:15.683439 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbbb2d19c66178b0c82659b2db6f18de3fabeb760dd451646705ad61d54e27e2\": container with ID starting with dbbb2d19c66178b0c82659b2db6f18de3fabeb760dd451646705ad61d54e27e2 not found: ID does not exist" containerID="dbbb2d19c66178b0c82659b2db6f18de3fabeb760dd451646705ad61d54e27e2" Oct 30 13:06:15.683556 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.683465 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbb2d19c66178b0c82659b2db6f18de3fabeb760dd451646705ad61d54e27e2"} err="failed to get container status \"dbbb2d19c66178b0c82659b2db6f18de3fabeb760dd451646705ad61d54e27e2\": rpc error: code = NotFound desc = could not find container \"dbbb2d19c66178b0c82659b2db6f18de3fabeb760dd451646705ad61d54e27e2\": container with ID starting with dbbb2d19c66178b0c82659b2db6f18de3fabeb760dd451646705ad61d54e27e2 not found: ID does not exist" Oct 30 13:06:15.716417 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.716378 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7c9b69-2356-49ba-ad43-2a8773f48a46-kube-api-access-6bsvg\" (UniqueName: \"kubernetes.io/projected/7c7c9b69-2356-49ba-ad43-2a8773f48a46-kube-api-access-6bsvg\") pod \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\" (UID: \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\") " Oct 30 13:06:15.716541 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.716502 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c7c9b69-2356-49ba-ad43-2a8773f48a46-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7c9b69-2356-49ba-ad43-2a8773f48a46-metrics-certs\") pod \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\" (UID: \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\") " Oct 30 13:06:15.716596 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.716545 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c7c9b69-2356-49ba-ad43-2a8773f48a46-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7c9b69-2356-49ba-ad43-2a8773f48a46-maas-api-tls\") pod \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\" (UID: \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\") " Oct 30 13:06:15.716702 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.716679 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c7c9b69-2356-49ba-ad43-2a8773f48a46-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7c9b69-2356-49ba-ad43-2a8773f48a46-openshift-service-ca\") pod \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\" (UID: \"7c7c9b69-2356-49ba-ad43-2a8773f48a46\") " Oct 30 13:06:15.717179 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.717134 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7c9b69-2356-49ba-ad43-2a8773f48a46-openshift-service-ca" pod "7c7c9b69-2356-49ba-ad43-2a8773f48a46" (UID: "7c7c9b69-2356-49ba-ad43-2a8773f48a46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:15.717336 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.717278 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7c9b69-2356-49ba-ad43-2a8773f48a46-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:15.718836 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.718812 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c7c9b69-2356-49ba-ad43-2a8773f48a46-metrics-certs" pod "7c7c9b69-2356-49ba-ad43-2a8773f48a46" (UID: "7c7c9b69-2356-49ba-ad43-2a8773f48a46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:15.718951 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.718842 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c7c9b69-2356-49ba-ad43-2a8773f48a46-maas-api-tls" pod "7c7c9b69-2356-49ba-ad43-2a8773f48a46" (UID: "7c7c9b69-2356-49ba-ad43-2a8773f48a46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:15.719023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.719006 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7c9b69-2356-49ba-ad43-2a8773f48a46-kube-api-access-6bsvg" pod "7c7c9b69-2356-49ba-ad43-2a8773f48a46" (UID: "7c7c9b69-2356-49ba-ad43-2a8773f48a46"). InnerVolumeSpecName "kube-api-access-6bsvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:15.818520 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.818480 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bsvg\" (UniqueName: \"kubernetes.io/projected/7c7c9b69-2356-49ba-ad43-2a8773f48a46-kube-api-access-6bsvg\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:15.818520 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.818521 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7c9b69-2356-49ba-ad43-2a8773f48a46-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:15.818729 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:15.818536 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7c9b69-2356-49ba-ad43-2a8773f48a46-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:16.064597 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:16.064563 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r"] Oct 30 13:06:16.076205 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:16.076175 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1333c8-bc5755cd4-5db9r"] Oct 30 13:06:16.282967 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:16.282871 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" podUID="6d9cb487-201b-4bc1-ba55-c3e7c4306174" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:06:17.091077 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:17.091030 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7c9b69-2356-49ba-ad43-2a8773f48a46" path="/var/lib/kubelet/pods/7c7c9b69-2356-49ba-ad43-2a8773f48a46/volumes" Oct 30 13:06:17.687568 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:17.687507 4468 generic.go:356] "Generic (PLEG): container finished" podID="6d9cb487-201b-4bc1-ba55-c3e7c4306174" containerID="7aa04a4d8bc3dfb0991a5bd27a2612dec6397e28f04ca4ffe0cf42f168c0739d" exitCode=0 Oct 30 13:06:17.687685 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:17.687585 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" event={"ID":"6d9cb487-201b-4bc1-ba55-c3e7c4306174","Type":"ContainerDied","Data":"7aa04a4d8bc3dfb0991a5bd27a2612dec6397e28f04ca4ffe0cf42f168c0739d"} Oct 30 13:06:17.687685 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:17.687640 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" event={"ID":"6d9cb487-201b-4bc1-ba55-c3e7c4306174","Type":"ContainerDied","Data":"3b6760955af4c3ccc0d51ff8d3e9213aa7a6f3651017bcdcec86989f8e97e53c"} Oct 30 13:06:17.687685 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:17.687657 4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b6760955af4c3ccc0d51ff8d3e9213aa7a6f3651017bcdcec86989f8e97e53c" Oct 30 13:06:17.700511 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:17.700493 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" Oct 30 13:06:17.843837 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:17.843812 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d9cb487-201b-4bc1-ba55-c3e7c4306174-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9cb487-201b-4bc1-ba55-c3e7c4306174-plugins-config-volume\") pod \"6d9cb487-201b-4bc1-ba55-c3e7c4306174\" (UID: \"6d9cb487-201b-4bc1-ba55-c3e7c4306174\") " Oct 30 13:06:17.843993 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:17.843906 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d9cb487-201b-4bc1-ba55-c3e7c4306174-kube-api-access-lh5zq\" (UniqueName: \"kubernetes.io/projected/6d9cb487-201b-4bc1-ba55-c3e7c4306174-kube-api-access-lh5zq\") pod \"6d9cb487-201b-4bc1-ba55-c3e7c4306174\" (UID: \"6d9cb487-201b-4bc1-ba55-c3e7c4306174\") " Oct 30 13:06:17.844283 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:17.844259 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d9cb487-201b-4bc1-ba55-c3e7c4306174-plugins-config-volume" pod "6d9cb487-201b-4bc1-ba55-c3e7c4306174" (UID: "6d9cb487-201b-4bc1-ba55-c3e7c4306174"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:17.845999 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:17.845976 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d9cb487-201b-4bc1-ba55-c3e7c4306174-kube-api-access-lh5zq" pod "6d9cb487-201b-4bc1-ba55-c3e7c4306174" (UID: "6d9cb487-201b-4bc1-ba55-c3e7c4306174"). InnerVolumeSpecName "kube-api-access-lh5zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:17.945924 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:17.945847 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9cb487-201b-4bc1-ba55-c3e7c4306174-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:17.945924 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:17.945873 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh5zq\" (UniqueName: \"kubernetes.io/projected/6d9cb487-201b-4bc1-ba55-c3e7c4306174-kube-api-access-lh5zq\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:18.419820 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:18.419783 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" podUID="64964ff4-80ce-4c89-af7b-19d3b24b07f4" containerName="payload-processing" containerID="cri-o://bbc6c33e039e8a82136c266fd96dbdbbdca18454df568a809ccd55c8ac1ea2f1" gracePeriod=25 Oct 30 13:06:18.427259 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:18.427228 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" podUID="64964ff4-80ce-4c89-af7b-19d3b24b07f4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:06:18.440414 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:18.440386 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" podUID="56adb9ab-1a6f-497c-8987-33123b8eb06a" containerName="payload-pre-processing" containerID="cri-o://3cbf3e7c21386a0e05a2f6e6f98d616851a35c3d1d7f856032ed485510ffbcb3" gracePeriod=25 Oct 30 13:06:18.693645 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:18.693555 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs" Oct 30 13:06:18.750004 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:18.749977 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs"] Oct 30 13:06:18.763491 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:18.763468 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1333c8-57c4665659-vmffs"] Oct 30 13:06:19.091387 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:19.091311 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d9cb487-201b-4bc1-ba55-c3e7c4306174" path="/var/lib/kubelet/pods/6d9cb487-201b-4bc1-ba55-c3e7c4306174/volumes" Oct 30 13:06:23.598634 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:06:23.598591 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:06:24.497772 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:24.497730 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" podUID="5db888ab-7750-41f1-9a4c-3add0f15cc6d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:06:24.497924 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:24.497780 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" podUID="bc9a08f6-1257-4e45-9a7d-9687d6554377" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:06:28.427138 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:28.427101 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" podUID="64964ff4-80ce-4c89-af7b-19d3b24b07f4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:06:28.427138 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:28.427121 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" podUID="56adb9ab-1a6f-497c-8987-33123b8eb06a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:06:34.498056 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:34.498013 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" podUID="bc9a08f6-1257-4e45-9a7d-9687d6554377" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:06:34.501819 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:34.501760 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" podUID="5db888ab-7750-41f1-9a4c-3add0f15cc6d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:06:35.261988 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.261962 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" Oct 30 13:06:35.322185 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.322152 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" Oct 30 13:06:35.331009 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.330295 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc9a08f6-1257-4e45-9a7d-9687d6554377-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9a08f6-1257-4e45-9a7d-9687d6554377-plugins-config-volume\") pod \"bc9a08f6-1257-4e45-9a7d-9687d6554377\" (UID: \"bc9a08f6-1257-4e45-9a7d-9687d6554377\") " Oct 30 13:06:35.331009 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.330516 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9a08f6-1257-4e45-9a7d-9687d6554377-kube-api-access-2bwxt\" (UniqueName: \"kubernetes.io/projected/bc9a08f6-1257-4e45-9a7d-9687d6554377-kube-api-access-2bwxt\") pod \"bc9a08f6-1257-4e45-9a7d-9687d6554377\" (UID: \"bc9a08f6-1257-4e45-9a7d-9687d6554377\") " Oct 30 13:06:35.332120 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.331829 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9a08f6-1257-4e45-9a7d-9687d6554377-plugins-config-volume" pod "bc9a08f6-1257-4e45-9a7d-9687d6554377" (UID: "bc9a08f6-1257-4e45-9a7d-9687d6554377"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:35.333095 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.332504 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9a08f6-1257-4e45-9a7d-9687d6554377-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:35.336104 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.334707 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9a08f6-1257-4e45-9a7d-9687d6554377-kube-api-access-2bwxt" pod "bc9a08f6-1257-4e45-9a7d-9687d6554377" (UID: "bc9a08f6-1257-4e45-9a7d-9687d6554377"). InnerVolumeSpecName "kube-api-access-2bwxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:35.433158 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.433127 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db888ab-7750-41f1-9a4c-3add0f15cc6d-kube-api-access-qbt2w\" (UniqueName: \"kubernetes.io/projected/5db888ab-7750-41f1-9a4c-3add0f15cc6d-kube-api-access-qbt2w\") pod \"5db888ab-7750-41f1-9a4c-3add0f15cc6d\" (UID: \"5db888ab-7750-41f1-9a4c-3add0f15cc6d\") " Oct 30 13:06:35.433335 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.433189 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db888ab-7750-41f1-9a4c-3add0f15cc6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db888ab-7750-41f1-9a4c-3add0f15cc6d-plugins-config-volume\") pod \"5db888ab-7750-41f1-9a4c-3add0f15cc6d\" (UID: \"5db888ab-7750-41f1-9a4c-3add0f15cc6d\") " Oct 30 13:06:35.433488 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.433472 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bwxt\" (UniqueName: \"kubernetes.io/projected/bc9a08f6-1257-4e45-9a7d-9687d6554377-kube-api-access-2bwxt\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:35.433619 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.433596 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db888ab-7750-41f1-9a4c-3add0f15cc6d-plugins-config-volume" pod "5db888ab-7750-41f1-9a4c-3add0f15cc6d" (UID: "5db888ab-7750-41f1-9a4c-3add0f15cc6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:35.435328 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.435301 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db888ab-7750-41f1-9a4c-3add0f15cc6d-kube-api-access-qbt2w" pod "5db888ab-7750-41f1-9a4c-3add0f15cc6d" (UID: "5db888ab-7750-41f1-9a4c-3add0f15cc6d"). InnerVolumeSpecName "kube-api-access-qbt2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:35.535177 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.535138 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbt2w\" (UniqueName: \"kubernetes.io/projected/5db888ab-7750-41f1-9a4c-3add0f15cc6d-kube-api-access-qbt2w\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:35.535177 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.535174 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db888ab-7750-41f1-9a4c-3add0f15cc6d-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:35.700871 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.700775 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk"] Oct 30 13:06:35.848905 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.848869 4468 generic.go:356] "Generic (PLEG): container finished" podID="bc9a08f6-1257-4e45-9a7d-9687d6554377" containerID="26c1b443cb3f7f1effe37bd0a04f9878fb15d08bfa54d811287c7dd53f089a87" exitCode=0 Oct 30 13:06:35.849092 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.848927 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" Oct 30 13:06:35.849092 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.848957 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" event={"ID":"bc9a08f6-1257-4e45-9a7d-9687d6554377","Type":"ContainerDied","Data":"26c1b443cb3f7f1effe37bd0a04f9878fb15d08bfa54d811287c7dd53f089a87"} Oct 30 13:06:35.849092 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.849014 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs" event={"ID":"bc9a08f6-1257-4e45-9a7d-9687d6554377","Type":"ContainerDied","Data":"2b61bef3798f96a691cdbdb5fe122836f67bf6246e0a4a79d8b0a880ec28945d"} Oct 30 13:06:35.849092 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.849042 4468 scope.go:122] "RemoveContainer" containerID="26c1b443cb3f7f1effe37bd0a04f9878fb15d08bfa54d811287c7dd53f089a87" Oct 30 13:06:35.850608 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.850586 4468 generic.go:356] "Generic (PLEG): container finished" podID="5db888ab-7750-41f1-9a4c-3add0f15cc6d" containerID="0dabc8ba218f01b32a71860b65add6d6de98fcb2ed635627e028295738d7e7b5" exitCode=0 Oct 30 13:06:35.850710 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.850641 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" Oct 30 13:06:35.850710 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.850678 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" event={"ID":"5db888ab-7750-41f1-9a4c-3add0f15cc6d","Type":"ContainerDied","Data":"0dabc8ba218f01b32a71860b65add6d6de98fcb2ed635627e028295738d7e7b5"} Oct 30 13:06:35.850819 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.850739 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2" event={"ID":"5db888ab-7750-41f1-9a4c-3add0f15cc6d","Type":"ContainerDied","Data":"4b91a306c006f50aa68f1efbd8afd3b1847ed923f49507beeb2c6d0ff9540dbb"} Oct 30 13:06:35.860222 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.860198 4468 scope.go:122] "RemoveContainer" containerID="26c1b443cb3f7f1effe37bd0a04f9878fb15d08bfa54d811287c7dd53f089a87" Oct 30 13:06:35.860540 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:06:35.860515 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c1b443cb3f7f1effe37bd0a04f9878fb15d08bfa54d811287c7dd53f089a87\": container with ID starting with 26c1b443cb3f7f1effe37bd0a04f9878fb15d08bfa54d811287c7dd53f089a87 not found: ID does not exist" containerID="26c1b443cb3f7f1effe37bd0a04f9878fb15d08bfa54d811287c7dd53f089a87" Oct 30 13:06:35.860642 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.860550 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c1b443cb3f7f1effe37bd0a04f9878fb15d08bfa54d811287c7dd53f089a87"} err="failed to get container status \"26c1b443cb3f7f1effe37bd0a04f9878fb15d08bfa54d811287c7dd53f089a87\": rpc error: code = NotFound desc = could not find container \"26c1b443cb3f7f1effe37bd0a04f9878fb15d08bfa54d811287c7dd53f089a87\": container with ID starting with 26c1b443cb3f7f1effe37bd0a04f9878fb15d08bfa54d811287c7dd53f089a87 not found: ID does not exist" Oct 30 13:06:35.860642 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.860575 4468 scope.go:122] "RemoveContainer" containerID="0dabc8ba218f01b32a71860b65add6d6de98fcb2ed635627e028295738d7e7b5" Oct 30 13:06:35.869406 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.869388 4468 scope.go:122] "RemoveContainer" containerID="0dabc8ba218f01b32a71860b65add6d6de98fcb2ed635627e028295738d7e7b5" Oct 30 13:06:35.869701 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:06:35.869679 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dabc8ba218f01b32a71860b65add6d6de98fcb2ed635627e028295738d7e7b5\": container with ID starting with 0dabc8ba218f01b32a71860b65add6d6de98fcb2ed635627e028295738d7e7b5 not found: ID does not exist" containerID="0dabc8ba218f01b32a71860b65add6d6de98fcb2ed635627e028295738d7e7b5" Oct 30 13:06:35.869804 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.869707 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dabc8ba218f01b32a71860b65add6d6de98fcb2ed635627e028295738d7e7b5"} err="failed to get container status \"0dabc8ba218f01b32a71860b65add6d6de98fcb2ed635627e028295738d7e7b5\": rpc error: code = NotFound desc = could not find container \"0dabc8ba218f01b32a71860b65add6d6de98fcb2ed635627e028295738d7e7b5\": container with ID starting with 0dabc8ba218f01b32a71860b65add6d6de98fcb2ed635627e028295738d7e7b5 not found: ID does not exist" Oct 30 13:06:35.924035 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.924001 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs"] Oct 30 13:06:35.928811 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.928780 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2545c7-5966c9ddb8-rjvgs"] Oct 30 13:06:35.952981 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.952917 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2"] Oct 30 13:06:35.957293 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:35.957271 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2545c7-6549664b79-f7qn2"] Oct 30 13:06:37.091217 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:37.091183 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db888ab-7750-41f1-9a4c-3add0f15cc6d" path="/var/lib/kubelet/pods/5db888ab-7750-41f1-9a4c-3add0f15cc6d/volumes" Oct 30 13:06:37.091620 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:37.091606 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc9a08f6-1257-4e45-9a7d-9687d6554377" path="/var/lib/kubelet/pods/bc9a08f6-1257-4e45-9a7d-9687d6554377/volumes" Oct 30 13:06:38.426874 ip-10-0-131-76 kubenswrapper[4468]: W1030 13:06:38.426849 4468 logging.go:55] [core] [Channel #22479 SubChannel #22480]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.203:50052", ServerName: "10.134.0.203:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.203:50052: connect: connection refused" Oct 30 13:06:38.427544 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.427516 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" podUID="56adb9ab-1a6f-497c-8987-33123b8eb06a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:06:38.703664 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.703633 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" Oct 30 13:06:38.743722 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.743700 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" Oct 30 13:06:38.773764 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.773741 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64964ff4-80ce-4c89-af7b-19d3b24b07f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64964ff4-80ce-4c89-af7b-19d3b24b07f4-plugins-config-volume\") pod \"64964ff4-80ce-4c89-af7b-19d3b24b07f4\" (UID: \"64964ff4-80ce-4c89-af7b-19d3b24b07f4\") " Oct 30 13:06:38.773895 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.773882 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64964ff4-80ce-4c89-af7b-19d3b24b07f4-kube-api-access-xpcrd\" (UniqueName: \"kubernetes.io/projected/64964ff4-80ce-4c89-af7b-19d3b24b07f4-kube-api-access-xpcrd\") pod \"64964ff4-80ce-4c89-af7b-19d3b24b07f4\" (UID: \"64964ff4-80ce-4c89-af7b-19d3b24b07f4\") " Oct 30 13:06:38.774220 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.774194 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64964ff4-80ce-4c89-af7b-19d3b24b07f4-plugins-config-volume" pod "64964ff4-80ce-4c89-af7b-19d3b24b07f4" (UID: "64964ff4-80ce-4c89-af7b-19d3b24b07f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:38.774330 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.774312 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64964ff4-80ce-4c89-af7b-19d3b24b07f4-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:38.776134 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.776111 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64964ff4-80ce-4c89-af7b-19d3b24b07f4-kube-api-access-xpcrd" pod "64964ff4-80ce-4c89-af7b-19d3b24b07f4" (UID: "64964ff4-80ce-4c89-af7b-19d3b24b07f4"). InnerVolumeSpecName "kube-api-access-xpcrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:38.874409 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.874378 4468 generic.go:356] "Generic (PLEG): container finished" podID="56adb9ab-1a6f-497c-8987-33123b8eb06a" containerID="3cbf3e7c21386a0e05a2f6e6f98d616851a35c3d1d7f856032ed485510ffbcb3" exitCode=0 Oct 30 13:06:38.874588 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.874426 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" Oct 30 13:06:38.874588 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.874461 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" event={"ID":"56adb9ab-1a6f-497c-8987-33123b8eb06a","Type":"ContainerDied","Data":"3cbf3e7c21386a0e05a2f6e6f98d616851a35c3d1d7f856032ed485510ffbcb3"} Oct 30 13:06:38.874588 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.874520 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22" event={"ID":"56adb9ab-1a6f-497c-8987-33123b8eb06a","Type":"ContainerDied","Data":"4c717a82462a50efaf7f90cbf0f5c57dd9f6caea664128875723fb625f3c996d"} Oct 30 13:06:38.874588 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.874549 4468 scope.go:122] "RemoveContainer" containerID="3cbf3e7c21386a0e05a2f6e6f98d616851a35c3d1d7f856032ed485510ffbcb3" Oct 30 13:06:38.874844 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.874822 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56adb9ab-1a6f-497c-8987-33123b8eb06a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56adb9ab-1a6f-497c-8987-33123b8eb06a-plugins-config-volume\") pod \"56adb9ab-1a6f-497c-8987-33123b8eb06a\" (UID: \"56adb9ab-1a6f-497c-8987-33123b8eb06a\") " Oct 30 13:06:38.874927 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.874913 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56adb9ab-1a6f-497c-8987-33123b8eb06a-kube-api-access-2dc4l\" (UniqueName: \"kubernetes.io/projected/56adb9ab-1a6f-497c-8987-33123b8eb06a-kube-api-access-2dc4l\") pod \"56adb9ab-1a6f-497c-8987-33123b8eb06a\" (UID: \"56adb9ab-1a6f-497c-8987-33123b8eb06a\") " Oct 30 13:06:38.875280 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.875253 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56adb9ab-1a6f-497c-8987-33123b8eb06a-plugins-config-volume" pod "56adb9ab-1a6f-497c-8987-33123b8eb06a" (UID: "56adb9ab-1a6f-497c-8987-33123b8eb06a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:38.875539 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.875519 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56adb9ab-1a6f-497c-8987-33123b8eb06a-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:38.875596 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.875543 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpcrd\" (UniqueName: \"kubernetes.io/projected/64964ff4-80ce-4c89-af7b-19d3b24b07f4-kube-api-access-xpcrd\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:38.876102 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.876048 4468 generic.go:356] "Generic (PLEG): container finished" podID="64964ff4-80ce-4c89-af7b-19d3b24b07f4" containerID="bbc6c33e039e8a82136c266fd96dbdbbdca18454df568a809ccd55c8ac1ea2f1" exitCode=0 Oct 30 13:06:38.876102 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.876091 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" event={"ID":"64964ff4-80ce-4c89-af7b-19d3b24b07f4","Type":"ContainerDied","Data":"bbc6c33e039e8a82136c266fd96dbdbbdca18454df568a809ccd55c8ac1ea2f1"} Oct 30 13:06:38.876269 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.876117 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" Oct 30 13:06:38.876269 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.876139 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" event={"ID":"64964ff4-80ce-4c89-af7b-19d3b24b07f4","Type":"ContainerDied","Data":"ddc664a80c141a6e81630cc081341063dbcd983124e507484e8f29c0e8acc90f"} Oct 30 13:06:38.877214 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.877193 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56adb9ab-1a6f-497c-8987-33123b8eb06a-kube-api-access-2dc4l" pod "56adb9ab-1a6f-497c-8987-33123b8eb06a" (UID: "56adb9ab-1a6f-497c-8987-33123b8eb06a"). InnerVolumeSpecName "kube-api-access-2dc4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:38.884671 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.884654 4468 scope.go:122] "RemoveContainer" containerID="3cbf3e7c21386a0e05a2f6e6f98d616851a35c3d1d7f856032ed485510ffbcb3" Oct 30 13:06:38.884940 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:06:38.884922 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cbf3e7c21386a0e05a2f6e6f98d616851a35c3d1d7f856032ed485510ffbcb3\": container with ID starting with 3cbf3e7c21386a0e05a2f6e6f98d616851a35c3d1d7f856032ed485510ffbcb3 not found: ID does not exist" containerID="3cbf3e7c21386a0e05a2f6e6f98d616851a35c3d1d7f856032ed485510ffbcb3" Oct 30 13:06:38.885002 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.884946 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cbf3e7c21386a0e05a2f6e6f98d616851a35c3d1d7f856032ed485510ffbcb3"} err="failed to get container status \"3cbf3e7c21386a0e05a2f6e6f98d616851a35c3d1d7f856032ed485510ffbcb3\": rpc error: code = NotFound desc = could not find container \"3cbf3e7c21386a0e05a2f6e6f98d616851a35c3d1d7f856032ed485510ffbcb3\": container with ID starting with 3cbf3e7c21386a0e05a2f6e6f98d616851a35c3d1d7f856032ed485510ffbcb3 not found: ID does not exist" Oct 30 13:06:38.885002 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.884966 4468 scope.go:122] "RemoveContainer" containerID="bbc6c33e039e8a82136c266fd96dbdbbdca18454df568a809ccd55c8ac1ea2f1" Oct 30 13:06:38.894320 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.894302 4468 scope.go:122] "RemoveContainer" containerID="bbc6c33e039e8a82136c266fd96dbdbbdca18454df568a809ccd55c8ac1ea2f1" Oct 30 13:06:38.894583 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:06:38.894563 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc6c33e039e8a82136c266fd96dbdbbdca18454df568a809ccd55c8ac1ea2f1\": container with ID starting with bbc6c33e039e8a82136c266fd96dbdbbdca18454df568a809ccd55c8ac1ea2f1 not found: ID does not exist" containerID="bbc6c33e039e8a82136c266fd96dbdbbdca18454df568a809ccd55c8ac1ea2f1" Oct 30 13:06:38.894648 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.894592 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc6c33e039e8a82136c266fd96dbdbbdca18454df568a809ccd55c8ac1ea2f1"} err="failed to get container status \"bbc6c33e039e8a82136c266fd96dbdbbdca18454df568a809ccd55c8ac1ea2f1\": rpc error: code = NotFound desc = could not find container \"bbc6c33e039e8a82136c266fd96dbdbbdca18454df568a809ccd55c8ac1ea2f1\": container with ID starting with bbc6c33e039e8a82136c266fd96dbdbbdca18454df568a809ccd55c8ac1ea2f1 not found: ID does not exist" Oct 30 13:06:38.932267 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.932212 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d"] Oct 30 13:06:38.937173 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.937154 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d"] Oct 30 13:06:38.976817 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:38.976785 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dc4l\" (UniqueName: \"kubernetes.io/projected/56adb9ab-1a6f-497c-8987-33123b8eb06a-kube-api-access-2dc4l\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:39.094945 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:39.093222 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64964ff4-80ce-4c89-af7b-19d3b24b07f4" path="/var/lib/kubelet/pods/64964ff4-80ce-4c89-af7b-19d3b24b07f4/volumes" Oct 30 13:06:39.218890 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:39.218821 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22"] Oct 30 13:06:39.222984 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:39.222963 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a786af-77cfc68667-zlw22"] Oct 30 13:06:39.426649 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:39.426617 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a786af-6665cd87c4-qhw7d" podUID="64964ff4-80ce-4c89-af7b-19d3b24b07f4" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.203:50052\" within 1s: context deadline exceeded" Oct 30 13:06:40.701848 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:40.701805 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" podUID="57f76bd7-cafb-48ad-9f99-8a7b88538a85" containerName="payload-pre-processing" containerID="cri-o://79c6e10614bdc667a132baa21bf3aed8416da9c91ed45f1ed6e01180d27d7972" gracePeriod=25 Oct 30 13:06:41.090546 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:41.090472 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56adb9ab-1a6f-497c-8987-33123b8eb06a" path="/var/lib/kubelet/pods/56adb9ab-1a6f-497c-8987-33123b8eb06a/volumes" Oct 30 13:06:45.694202 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:45.694160 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" podUID="57f76bd7-cafb-48ad-9f99-8a7b88538a85" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:06:45.904272 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:45.904237 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8"] Oct 30 13:06:50.456371 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.456330 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw"] Oct 30 13:06:50.457280 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.456634 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" podUID="2b10586a-875c-4812-9bd2-0e65b81d0224" containerName="maas-api" containerID="cri-o://230d8431a796d2c44e22ead60a63c44dcabfa7b00db2d7c1ae4bda649be239e4" gracePeriod=30 Oct 30 13:06:50.769091 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.769051 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" Oct 30 13:06:50.819799 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.819711 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb"] Oct 30 13:06:50.820275 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.819994 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" podUID="bcf35b28-b756-4281-80d7-bc361b2d56e3" containerName="maas-api" containerID="cri-o://0eae3f80cdcd2815da50353fd8b07348ac0731d97aabb33f4696860c2dc4aeea" gracePeriod=30 Oct 30 13:06:50.903457 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.903420 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b10586a-875c-4812-9bd2-0e65b81d0224-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b10586a-875c-4812-9bd2-0e65b81d0224-openshift-service-ca\") pod \"2b10586a-875c-4812-9bd2-0e65b81d0224\" (UID: \"2b10586a-875c-4812-9bd2-0e65b81d0224\") " Oct 30 13:06:50.903653 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.903634 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b10586a-875c-4812-9bd2-0e65b81d0224-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b10586a-875c-4812-9bd2-0e65b81d0224-metrics-certs\") pod \"2b10586a-875c-4812-9bd2-0e65b81d0224\" (UID: \"2b10586a-875c-4812-9bd2-0e65b81d0224\") " Oct 30 13:06:50.903719 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.903676 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b10586a-875c-4812-9bd2-0e65b81d0224-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b10586a-875c-4812-9bd2-0e65b81d0224-maas-api-tls\") pod \"2b10586a-875c-4812-9bd2-0e65b81d0224\" (UID: \"2b10586a-875c-4812-9bd2-0e65b81d0224\") " Oct 30 13:06:50.903880 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.903848 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b10586a-875c-4812-9bd2-0e65b81d0224-openshift-service-ca" pod "2b10586a-875c-4812-9bd2-0e65b81d0224" (UID: "2b10586a-875c-4812-9bd2-0e65b81d0224"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:50.904215 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.904189 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b10586a-875c-4812-9bd2-0e65b81d0224-kube-api-access-sv94k\" (UniqueName: \"kubernetes.io/projected/2b10586a-875c-4812-9bd2-0e65b81d0224-kube-api-access-sv94k\") pod \"2b10586a-875c-4812-9bd2-0e65b81d0224\" (UID: \"2b10586a-875c-4812-9bd2-0e65b81d0224\") " Oct 30 13:06:50.904531 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.904494 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" podUID="69e26174-f25d-4904-b2d8-b69a962ca344" containerName="payload-pre-processing" containerID="cri-o://bc69f25f39073c9797f586e24247e937137594b2d4e1ff4b2170d8a9ee0c9cfd" gracePeriod=25 Oct 30 13:06:50.904713 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.904689 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b10586a-875c-4812-9bd2-0e65b81d0224-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:50.906780 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.906751 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b10586a-875c-4812-9bd2-0e65b81d0224-metrics-certs" pod "2b10586a-875c-4812-9bd2-0e65b81d0224" (UID: "2b10586a-875c-4812-9bd2-0e65b81d0224"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:50.906941 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.906908 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b10586a-875c-4812-9bd2-0e65b81d0224-maas-api-tls" pod "2b10586a-875c-4812-9bd2-0e65b81d0224" (UID: "2b10586a-875c-4812-9bd2-0e65b81d0224"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:50.907164 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.907044 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b10586a-875c-4812-9bd2-0e65b81d0224-kube-api-access-sv94k" pod "2b10586a-875c-4812-9bd2-0e65b81d0224" (UID: "2b10586a-875c-4812-9bd2-0e65b81d0224"). InnerVolumeSpecName "kube-api-access-sv94k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:50.953121 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.953088 4468 generic.go:356] "Generic (PLEG): container finished" podID="2b10586a-875c-4812-9bd2-0e65b81d0224" containerID="230d8431a796d2c44e22ead60a63c44dcabfa7b00db2d7c1ae4bda649be239e4" exitCode=0 Oct 30 13:06:50.953245 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.953151 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" Oct 30 13:06:50.953245 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.953166 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" event={"ID":"2b10586a-875c-4812-9bd2-0e65b81d0224","Type":"ContainerDied","Data":"230d8431a796d2c44e22ead60a63c44dcabfa7b00db2d7c1ae4bda649be239e4"} Oct 30 13:06:50.953245 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.953227 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw" event={"ID":"2b10586a-875c-4812-9bd2-0e65b81d0224","Type":"ContainerDied","Data":"d583b58500fd08fab0afddde676e78c91545c571ce0ae07c2283140ac7d9fc19"} Oct 30 13:06:50.953418 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.953258 4468 scope.go:122] "RemoveContainer" containerID="230d8431a796d2c44e22ead60a63c44dcabfa7b00db2d7c1ae4bda649be239e4" Oct 30 13:06:50.955299 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.955272 4468 generic.go:356] "Generic (PLEG): container finished" podID="bcf35b28-b756-4281-80d7-bc361b2d56e3" containerID="0eae3f80cdcd2815da50353fd8b07348ac0731d97aabb33f4696860c2dc4aeea" exitCode=0 Oct 30 13:06:50.955408 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.955341 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" event={"ID":"bcf35b28-b756-4281-80d7-bc361b2d56e3","Type":"ContainerDied","Data":"0eae3f80cdcd2815da50353fd8b07348ac0731d97aabb33f4696860c2dc4aeea"} Oct 30 13:06:50.964545 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.964528 4468 scope.go:122] "RemoveContainer" containerID="230d8431a796d2c44e22ead60a63c44dcabfa7b00db2d7c1ae4bda649be239e4" Oct 30 13:06:50.964931 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:06:50.964912 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"230d8431a796d2c44e22ead60a63c44dcabfa7b00db2d7c1ae4bda649be239e4\": container with ID starting with 230d8431a796d2c44e22ead60a63c44dcabfa7b00db2d7c1ae4bda649be239e4 not found: ID does not exist" containerID="230d8431a796d2c44e22ead60a63c44dcabfa7b00db2d7c1ae4bda649be239e4" Oct 30 13:06:50.964985 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:50.964941 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"230d8431a796d2c44e22ead60a63c44dcabfa7b00db2d7c1ae4bda649be239e4"} err="failed to get container status \"230d8431a796d2c44e22ead60a63c44dcabfa7b00db2d7c1ae4bda649be239e4\": rpc error: code = NotFound desc = could not find container \"230d8431a796d2c44e22ead60a63c44dcabfa7b00db2d7c1ae4bda649be239e4\": container with ID starting with 230d8431a796d2c44e22ead60a63c44dcabfa7b00db2d7c1ae4bda649be239e4 not found: ID does not exist" Oct 30 13:06:51.008102 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.008032 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv94k\" (UniqueName: \"kubernetes.io/projected/2b10586a-875c-4812-9bd2-0e65b81d0224-kube-api-access-sv94k\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:51.008102 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.008085 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b10586a-875c-4812-9bd2-0e65b81d0224-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:51.008102 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.008102 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b10586a-875c-4812-9bd2-0e65b81d0224-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:51.017742 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.017716 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw"] Oct 30 13:06:51.023871 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.023846 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dd0628-7986f46876-r55zw"] Oct 30 13:06:51.083037 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.083013 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" Oct 30 13:06:51.090590 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.090562 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b10586a-875c-4812-9bd2-0e65b81d0224" path="/var/lib/kubelet/pods/2b10586a-875c-4812-9bd2-0e65b81d0224/volumes" Oct 30 13:06:51.210888 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.210851 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcf35b28-b756-4281-80d7-bc361b2d56e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf35b28-b756-4281-80d7-bc361b2d56e3-metrics-certs\") pod \"bcf35b28-b756-4281-80d7-bc361b2d56e3\" (UID: \"bcf35b28-b756-4281-80d7-bc361b2d56e3\") " Oct 30 13:06:51.211099 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.211047 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcf35b28-b756-4281-80d7-bc361b2d56e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf35b28-b756-4281-80d7-bc361b2d56e3-openshift-service-ca\") pod \"bcf35b28-b756-4281-80d7-bc361b2d56e3\" (UID: \"bcf35b28-b756-4281-80d7-bc361b2d56e3\") " Oct 30 13:06:51.211182 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.211128 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf35b28-b756-4281-80d7-bc361b2d56e3-kube-api-access-dxhfk\" (UniqueName: \"kubernetes.io/projected/bcf35b28-b756-4281-80d7-bc361b2d56e3-kube-api-access-dxhfk\") pod \"bcf35b28-b756-4281-80d7-bc361b2d56e3\" (UID: \"bcf35b28-b756-4281-80d7-bc361b2d56e3\") " Oct 30 13:06:51.211182 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.211168 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcf35b28-b756-4281-80d7-bc361b2d56e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcf35b28-b756-4281-80d7-bc361b2d56e3-maas-api-tls\") pod \"bcf35b28-b756-4281-80d7-bc361b2d56e3\" (UID: \"bcf35b28-b756-4281-80d7-bc361b2d56e3\") " Oct 30 13:06:51.211497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.211467 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf35b28-b756-4281-80d7-bc361b2d56e3-openshift-service-ca" pod "bcf35b28-b756-4281-80d7-bc361b2d56e3" (UID: "bcf35b28-b756-4281-80d7-bc361b2d56e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:51.211911 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.211783 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf35b28-b756-4281-80d7-bc361b2d56e3-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:51.213280 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.213246 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf35b28-b756-4281-80d7-bc361b2d56e3-kube-api-access-dxhfk" pod "bcf35b28-b756-4281-80d7-bc361b2d56e3" (UID: "bcf35b28-b756-4281-80d7-bc361b2d56e3"). InnerVolumeSpecName "kube-api-access-dxhfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:51.213509 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.213482 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf35b28-b756-4281-80d7-bc361b2d56e3-metrics-certs" pod "bcf35b28-b756-4281-80d7-bc361b2d56e3" (UID: "bcf35b28-b756-4281-80d7-bc361b2d56e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:51.213509 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.213495 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf35b28-b756-4281-80d7-bc361b2d56e3-maas-api-tls" pod "bcf35b28-b756-4281-80d7-bc361b2d56e3" (UID: "bcf35b28-b756-4281-80d7-bc361b2d56e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:51.313099 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.312957 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf35b28-b756-4281-80d7-bc361b2d56e3-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:51.313099 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.312998 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxhfk\" (UniqueName: \"kubernetes.io/projected/bcf35b28-b756-4281-80d7-bc361b2d56e3-kube-api-access-dxhfk\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:51.313099 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.313007 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcf35b28-b756-4281-80d7-bc361b2d56e3-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:06:51.964407 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.964378 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" Oct 30 13:06:51.964407 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.964389 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb" event={"ID":"bcf35b28-b756-4281-80d7-bc361b2d56e3","Type":"ContainerDied","Data":"830fd204a0bce340a80995f3872b82dc7b2f28a0c7af02a7eefb373596ac513b"} Oct 30 13:06:51.964907 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:51.964441 4468 scope.go:122] "RemoveContainer" containerID="0eae3f80cdcd2815da50353fd8b07348ac0731d97aabb33f4696860c2dc4aeea" Oct 30 13:06:52.032256 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:52.032230 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb"] Oct 30 13:06:52.045398 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:52.045374 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c3f12-6f5b5d7b4d-z9thb"] Oct 30 13:06:52.297809 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:52.297733 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" podUID="69e26174-f25d-4904-b2d8-b69a962ca344" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:06:52.902292 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:52.902251 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv"] Oct 30 13:06:53.090453 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:53.090409 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf35b28-b756-4281-80d7-bc361b2d56e3" path="/var/lib/kubelet/pods/bcf35b28-b756-4281-80d7-bc361b2d56e3/volumes" Oct 30 13:06:53.599792 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:06:53.599753 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:06:55.693630 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:55.693587 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" podUID="57f76bd7-cafb-48ad-9f99-8a7b88538a85" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:06:57.903577 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:06:57.903540 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" podUID="110bd521-953f-48c7-990a-823b164a281b" containerName="payload-pre-processing" containerID="cri-o://22e689d84524c87a0d40c8b99895125f26a3aa09d97442cf9b2900e291ae7dc5" gracePeriod=25 Oct 30 13:07:00.010738 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:00.010704 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb"] Oct 30 13:07:00.011226 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:00.010929 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" podUID="af770c09-3be0-418b-8737-3cd556b08035" containerName="maas-api" containerID="cri-o://fca61f1217195aa778e20fec43b044864ed365b94fd28e920623988393f4ee93" gracePeriod=30 Oct 30 13:07:00.298548 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:00.298478 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" Oct 30 13:07:00.397750 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:00.397715 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af770c09-3be0-418b-8737-3cd556b08035-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af770c09-3be0-418b-8737-3cd556b08035-maas-api-tls\") pod \"af770c09-3be0-418b-8737-3cd556b08035\" (UID: \"af770c09-3be0-418b-8737-3cd556b08035\") " Oct 30 13:07:00.397897 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:00.397764 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af770c09-3be0-418b-8737-3cd556b08035-kube-api-access-2n59p\" (UniqueName: \"kubernetes.io/projected/af770c09-3be0-418b-8737-3cd556b08035-kube-api-access-2n59p\") pod \"af770c09-3be0-418b-8737-3cd556b08035\" (UID: \"af770c09-3be0-418b-8737-3cd556b08035\") " Oct 30 13:07:00.397897 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:00.397794 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af770c09-3be0-418b-8737-3cd556b08035-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af770c09-3be0-418b-8737-3cd556b08035-metrics-certs\") pod \"af770c09-3be0-418b-8737-3cd556b08035\" (UID: \"af770c09-3be0-418b-8737-3cd556b08035\") " Oct 30 13:07:00.397897 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:00.397861 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af770c09-3be0-418b-8737-3cd556b08035-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af770c09-3be0-418b-8737-3cd556b08035-openshift-service-ca\") pod \"af770c09-3be0-418b-8737-3cd556b08035\" (UID: \"af770c09-3be0-418b-8737-3cd556b08035\") " Oct 30 13:07:00.398381 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:00.398345 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af770c09-3be0-418b-8737-3cd556b08035-openshift-service-ca" pod "af770c09-3be0-418b-8737-3cd556b08035" (UID: "af770c09-3be0-418b-8737-3cd556b08035"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:00.400006 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:00.399971 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af770c09-3be0-418b-8737-3cd556b08035-metrics-certs" pod "af770c09-3be0-418b-8737-3cd556b08035" (UID: "af770c09-3be0-418b-8737-3cd556b08035"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:00.400352 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:00.400331 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af770c09-3be0-418b-8737-3cd556b08035-kube-api-access-2n59p" pod "af770c09-3be0-418b-8737-3cd556b08035" (UID: "af770c09-3be0-418b-8737-3cd556b08035"). InnerVolumeSpecName "kube-api-access-2n59p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:00.400428 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:00.400365 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af770c09-3be0-418b-8737-3cd556b08035-maas-api-tls" pod "af770c09-3be0-418b-8737-3cd556b08035" (UID: "af770c09-3be0-418b-8737-3cd556b08035"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:00.499472 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:00.499376 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af770c09-3be0-418b-8737-3cd556b08035-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:00.499472 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:00.499417 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n59p\" (UniqueName: \"kubernetes.io/projected/af770c09-3be0-418b-8737-3cd556b08035-kube-api-access-2n59p\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:00.499472 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:00.499433 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af770c09-3be0-418b-8737-3cd556b08035-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:00.499472 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:00.499447 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af770c09-3be0-418b-8737-3cd556b08035-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:00.954550 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:00.954523 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" Oct 30 13:07:01.007166 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.007058 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57f76bd7-cafb-48ad-9f99-8a7b88538a85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f76bd7-cafb-48ad-9f99-8a7b88538a85-plugins-config-volume\") pod \"57f76bd7-cafb-48ad-9f99-8a7b88538a85\" (UID: \"57f76bd7-cafb-48ad-9f99-8a7b88538a85\") " Oct 30 13:07:01.007166 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.007139 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f76bd7-cafb-48ad-9f99-8a7b88538a85-kube-api-access-75rf6\" (UniqueName: \"kubernetes.io/projected/57f76bd7-cafb-48ad-9f99-8a7b88538a85-kube-api-access-75rf6\") pod \"57f76bd7-cafb-48ad-9f99-8a7b88538a85\" (UID: \"57f76bd7-cafb-48ad-9f99-8a7b88538a85\") " Oct 30 13:07:01.008283 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.008255 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f76bd7-cafb-48ad-9f99-8a7b88538a85-plugins-config-volume" pod "57f76bd7-cafb-48ad-9f99-8a7b88538a85" (UID: "57f76bd7-cafb-48ad-9f99-8a7b88538a85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:01.009873 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.009837 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f76bd7-cafb-48ad-9f99-8a7b88538a85-kube-api-access-75rf6" pod "57f76bd7-cafb-48ad-9f99-8a7b88538a85" (UID: "57f76bd7-cafb-48ad-9f99-8a7b88538a85"). InnerVolumeSpecName "kube-api-access-75rf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:01.025780 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.025747 4468 generic.go:356] "Generic (PLEG): container finished" podID="af770c09-3be0-418b-8737-3cd556b08035" containerID="fca61f1217195aa778e20fec43b044864ed365b94fd28e920623988393f4ee93" exitCode=0 Oct 30 13:07:01.026387 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.025793 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" Oct 30 13:07:01.026387 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.025827 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" event={"ID":"af770c09-3be0-418b-8737-3cd556b08035","Type":"ContainerDied","Data":"fca61f1217195aa778e20fec43b044864ed365b94fd28e920623988393f4ee93"} Oct 30 13:07:01.026387 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.025920 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb" event={"ID":"af770c09-3be0-418b-8737-3cd556b08035","Type":"ContainerDied","Data":"e6b0818128dc3fd464a264423f33ef36951873ceac406d238ee01ff0d4f31712"} Oct 30 13:07:01.026387 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.025948 4468 scope.go:122] "RemoveContainer" containerID="fca61f1217195aa778e20fec43b044864ed365b94fd28e920623988393f4ee93" Oct 30 13:07:01.029872 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.029844 4468 generic.go:356] "Generic (PLEG): container finished" podID="57f76bd7-cafb-48ad-9f99-8a7b88538a85" containerID="79c6e10614bdc667a132baa21bf3aed8416da9c91ed45f1ed6e01180d27d7972" exitCode=0 Oct 30 13:07:01.029987 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.029907 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" event={"ID":"57f76bd7-cafb-48ad-9f99-8a7b88538a85","Type":"ContainerDied","Data":"79c6e10614bdc667a132baa21bf3aed8416da9c91ed45f1ed6e01180d27d7972"} Oct 30 13:07:01.029987 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.029947 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" event={"ID":"57f76bd7-cafb-48ad-9f99-8a7b88538a85","Type":"ContainerDied","Data":"2be808856d113fbd6a0c5d7ed6daf04e7a6a1d6e6a13bfc6780e393af4f79b9b"} Oct 30 13:07:01.030156 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.030012 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk" Oct 30 13:07:01.049526 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.049501 4468 scope.go:122] "RemoveContainer" containerID="fca61f1217195aa778e20fec43b044864ed365b94fd28e920623988393f4ee93" Oct 30 13:07:01.049839 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:07:01.049818 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca61f1217195aa778e20fec43b044864ed365b94fd28e920623988393f4ee93\": container with ID starting with fca61f1217195aa778e20fec43b044864ed365b94fd28e920623988393f4ee93 not found: ID does not exist" containerID="fca61f1217195aa778e20fec43b044864ed365b94fd28e920623988393f4ee93" Oct 30 13:07:01.049940 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.049856 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca61f1217195aa778e20fec43b044864ed365b94fd28e920623988393f4ee93"} err="failed to get container status \"fca61f1217195aa778e20fec43b044864ed365b94fd28e920623988393f4ee93\": rpc error: code = NotFound desc = could not find container \"fca61f1217195aa778e20fec43b044864ed365b94fd28e920623988393f4ee93\": container with ID starting with fca61f1217195aa778e20fec43b044864ed365b94fd28e920623988393f4ee93 not found: ID does not exist" Oct 30 13:07:01.049940 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.049879 4468 scope.go:122] "RemoveContainer" containerID="79c6e10614bdc667a132baa21bf3aed8416da9c91ed45f1ed6e01180d27d7972" Oct 30 13:07:01.058849 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.058812 4468 scope.go:122] "RemoveContainer" containerID="79c6e10614bdc667a132baa21bf3aed8416da9c91ed45f1ed6e01180d27d7972" Oct 30 13:07:01.059113 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:07:01.059092 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79c6e10614bdc667a132baa21bf3aed8416da9c91ed45f1ed6e01180d27d7972\": container with ID starting with 79c6e10614bdc667a132baa21bf3aed8416da9c91ed45f1ed6e01180d27d7972 not found: ID does not exist" containerID="79c6e10614bdc667a132baa21bf3aed8416da9c91ed45f1ed6e01180d27d7972" Oct 30 13:07:01.059178 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.059123 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79c6e10614bdc667a132baa21bf3aed8416da9c91ed45f1ed6e01180d27d7972"} err="failed to get container status \"79c6e10614bdc667a132baa21bf3aed8416da9c91ed45f1ed6e01180d27d7972\": rpc error: code = NotFound desc = could not find container \"79c6e10614bdc667a132baa21bf3aed8416da9c91ed45f1ed6e01180d27d7972\": container with ID starting with 79c6e10614bdc667a132baa21bf3aed8416da9c91ed45f1ed6e01180d27d7972 not found: ID does not exist" Oct 30 13:07:01.092766 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.092745 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb"] Oct 30 13:07:01.096477 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.096457 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f7d14-66f597779b-bwgzb"] Oct 30 13:07:01.108623 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.108602 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75rf6\" (UniqueName: \"kubernetes.io/projected/57f76bd7-cafb-48ad-9f99-8a7b88538a85-kube-api-access-75rf6\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:01.108623 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.108624 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f76bd7-cafb-48ad-9f99-8a7b88538a85-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:01.134997 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.134978 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk"] Oct 30 13:07:01.138166 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:01.138145 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c3f12-549d99f87-kxbqk"] Oct 30 13:07:02.298157 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:02.298119 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" podUID="69e26174-f25d-4904-b2d8-b69a962ca344" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:02.428271 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:02.428236 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" podUID="110bd521-953f-48c7-990a-823b164a281b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:03.089912 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:03.089881 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57f76bd7-cafb-48ad-9f99-8a7b88538a85" path="/var/lib/kubelet/pods/57f76bd7-cafb-48ad-9f99-8a7b88538a85/volumes" Oct 30 13:07:03.090501 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:03.090482 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af770c09-3be0-418b-8737-3cd556b08035" path="/var/lib/kubelet/pods/af770c09-3be0-418b-8737-3cd556b08035/volumes" Oct 30 13:07:04.407703 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:04.407671 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6"] Oct 30 13:07:04.421562 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:04.421534 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z"] Oct 30 13:07:05.636720 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:05.636681 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm"] Oct 30 13:07:05.637267 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:05.637001 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" podUID="a880cac8-149c-42bc-87c1-31464bf24279" containerName="maas-api" containerID="cri-o://824134fccb9e1550302b953bc9637d7d1fb18cd72e70ef2138a7da4e4223bf55" gracePeriod=30 Oct 30 13:07:05.908853 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:05.908821 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" Oct 30 13:07:05.968562 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:05.968534 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a880cac8-149c-42bc-87c1-31464bf24279-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a880cac8-149c-42bc-87c1-31464bf24279-openshift-service-ca\") pod \"a880cac8-149c-42bc-87c1-31464bf24279\" (UID: \"a880cac8-149c-42bc-87c1-31464bf24279\") " Oct 30 13:07:05.968742 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:05.968677 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a880cac8-149c-42bc-87c1-31464bf24279-kube-api-access-6m82k\" (UniqueName: \"kubernetes.io/projected/a880cac8-149c-42bc-87c1-31464bf24279-kube-api-access-6m82k\") pod \"a880cac8-149c-42bc-87c1-31464bf24279\" (UID: \"a880cac8-149c-42bc-87c1-31464bf24279\") " Oct 30 13:07:05.968742 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:05.968723 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a880cac8-149c-42bc-87c1-31464bf24279-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a880cac8-149c-42bc-87c1-31464bf24279-metrics-certs\") pod \"a880cac8-149c-42bc-87c1-31464bf24279\" (UID: \"a880cac8-149c-42bc-87c1-31464bf24279\") " Oct 30 13:07:05.968866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:05.968769 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a880cac8-149c-42bc-87c1-31464bf24279-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a880cac8-149c-42bc-87c1-31464bf24279-maas-api-tls\") pod \"a880cac8-149c-42bc-87c1-31464bf24279\" (UID: \"a880cac8-149c-42bc-87c1-31464bf24279\") " Oct 30 13:07:05.968975 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:05.968945 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a880cac8-149c-42bc-87c1-31464bf24279-openshift-service-ca" pod "a880cac8-149c-42bc-87c1-31464bf24279" (UID: "a880cac8-149c-42bc-87c1-31464bf24279"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:05.969332 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:05.969303 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a880cac8-149c-42bc-87c1-31464bf24279-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:05.971274 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:05.971133 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a880cac8-149c-42bc-87c1-31464bf24279-kube-api-access-6m82k" pod "a880cac8-149c-42bc-87c1-31464bf24279" (UID: "a880cac8-149c-42bc-87c1-31464bf24279"). InnerVolumeSpecName "kube-api-access-6m82k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:05.971274 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:05.971238 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a880cac8-149c-42bc-87c1-31464bf24279-metrics-certs" pod "a880cac8-149c-42bc-87c1-31464bf24279" (UID: "a880cac8-149c-42bc-87c1-31464bf24279"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:05.978530 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:05.978504 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a880cac8-149c-42bc-87c1-31464bf24279-maas-api-tls" pod "a880cac8-149c-42bc-87c1-31464bf24279" (UID: "a880cac8-149c-42bc-87c1-31464bf24279"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:06.063147 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:06.063116 4468 generic.go:356] "Generic (PLEG): container finished" podID="a880cac8-149c-42bc-87c1-31464bf24279" containerID="824134fccb9e1550302b953bc9637d7d1fb18cd72e70ef2138a7da4e4223bf55" exitCode=0 Oct 30 13:07:06.063395 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:06.063184 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" event={"ID":"a880cac8-149c-42bc-87c1-31464bf24279","Type":"ContainerDied","Data":"824134fccb9e1550302b953bc9637d7d1fb18cd72e70ef2138a7da4e4223bf55"} Oct 30 13:07:06.063541 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:06.063433 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" event={"ID":"a880cac8-149c-42bc-87c1-31464bf24279","Type":"ContainerDied","Data":"a6a1f382205360884dc896eb50d1daf94413cba53d217de13026e0d41822475a"} Oct 30 13:07:06.063541 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:06.063468 4468 scope.go:122] "RemoveContainer" containerID="824134fccb9e1550302b953bc9637d7d1fb18cd72e70ef2138a7da4e4223bf55" Oct 30 13:07:06.063541 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:06.063265 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" Oct 30 13:07:06.075933 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:06.075900 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m82k\" (UniqueName: \"kubernetes.io/projected/a880cac8-149c-42bc-87c1-31464bf24279-kube-api-access-6m82k\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:06.075933 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:06.075926 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a880cac8-149c-42bc-87c1-31464bf24279-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:06.075933 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:06.075937 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a880cac8-149c-42bc-87c1-31464bf24279-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:06.098083 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:06.098045 4468 scope.go:122] "RemoveContainer" containerID="824134fccb9e1550302b953bc9637d7d1fb18cd72e70ef2138a7da4e4223bf55" Oct 30 13:07:06.098537 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:07:06.098489 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"824134fccb9e1550302b953bc9637d7d1fb18cd72e70ef2138a7da4e4223bf55\": container with ID starting with 824134fccb9e1550302b953bc9637d7d1fb18cd72e70ef2138a7da4e4223bf55 not found: ID does not exist" containerID="824134fccb9e1550302b953bc9637d7d1fb18cd72e70ef2138a7da4e4223bf55" Oct 30 13:07:06.098708 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:06.098676 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"824134fccb9e1550302b953bc9637d7d1fb18cd72e70ef2138a7da4e4223bf55"} err="failed to get container status \"824134fccb9e1550302b953bc9637d7d1fb18cd72e70ef2138a7da4e4223bf55\": rpc error: code = NotFound desc = could not find container \"824134fccb9e1550302b953bc9637d7d1fb18cd72e70ef2138a7da4e4223bf55\": container with ID starting with 824134fccb9e1550302b953bc9637d7d1fb18cd72e70ef2138a7da4e4223bf55 not found: ID does not exist" Oct 30 13:07:06.126277 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:06.126240 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm"] Oct 30 13:07:06.130762 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:06.130732 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm"] Oct 30 13:07:07.089761 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:07.089730 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a880cac8-149c-42bc-87c1-31464bf24279" path="/var/lib/kubelet/pods/a880cac8-149c-42bc-87c1-31464bf24279/volumes" Oct 30 13:07:08.821891 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:08.821852 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12fd33-67dcb59c87-qrfsm" podUID="a880cac8-149c-42bc-87c1-31464bf24279" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.219:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:10.799601 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:10.799564 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4"] Oct 30 13:07:10.846992 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:10.846958 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2"] Oct 30 13:07:11.097601 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:11.097508 4468 generic.go:356] "Generic (PLEG): container finished" podID="69e26174-f25d-4904-b2d8-b69a962ca344" containerID="bc69f25f39073c9797f586e24247e937137594b2d4e1ff4b2170d8a9ee0c9cfd" exitCode=0 Oct 30 13:07:11.097774 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:11.097591 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" event={"ID":"69e26174-f25d-4904-b2d8-b69a962ca344","Type":"ContainerDied","Data":"bc69f25f39073c9797f586e24247e937137594b2d4e1ff4b2170d8a9ee0c9cfd"} Oct 30 13:07:11.603929 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:11.603901 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" Oct 30 13:07:11.718635 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:11.718565 4468 scope.go:122] "RemoveContainer" containerID="7aa04a4d8bc3dfb0991a5bd27a2612dec6397e28f04ca4ffe0cf42f168c0739d" Oct 30 13:07:11.728257 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:11.728236 4468 scope.go:122] "RemoveContainer" containerID="bc69f25f39073c9797f586e24247e937137594b2d4e1ff4b2170d8a9ee0c9cfd" Oct 30 13:07:11.757252 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:11.757232 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e26174-f25d-4904-b2d8-b69a962ca344-kube-api-access-9kr22\" (UniqueName: \"kubernetes.io/projected/69e26174-f25d-4904-b2d8-b69a962ca344-kube-api-access-9kr22\") pod \"69e26174-f25d-4904-b2d8-b69a962ca344\" (UID: \"69e26174-f25d-4904-b2d8-b69a962ca344\") " Oct 30 13:07:11.757321 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:11.757278 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69e26174-f25d-4904-b2d8-b69a962ca344-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e26174-f25d-4904-b2d8-b69a962ca344-plugins-config-volume\") pod \"69e26174-f25d-4904-b2d8-b69a962ca344\" (UID: \"69e26174-f25d-4904-b2d8-b69a962ca344\") " Oct 30 13:07:11.757650 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:11.757622 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e26174-f25d-4904-b2d8-b69a962ca344-plugins-config-volume" pod "69e26174-f25d-4904-b2d8-b69a962ca344" (UID: "69e26174-f25d-4904-b2d8-b69a962ca344"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:11.757933 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:11.757916 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e26174-f25d-4904-b2d8-b69a962ca344-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:11.759317 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:11.759292 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e26174-f25d-4904-b2d8-b69a962ca344-kube-api-access-9kr22" pod "69e26174-f25d-4904-b2d8-b69a962ca344" (UID: "69e26174-f25d-4904-b2d8-b69a962ca344"). InnerVolumeSpecName "kube-api-access-9kr22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:11.858795 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:11.858764 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kr22\" (UniqueName: \"kubernetes.io/projected/69e26174-f25d-4904-b2d8-b69a962ca344-kube-api-access-9kr22\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:12.103265 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:12.103191 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" Oct 30 13:07:12.103400 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:12.103193 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8" event={"ID":"69e26174-f25d-4904-b2d8-b69a962ca344","Type":"ContainerDied","Data":"57765afcb0954b25c4d13e9168e6539d4b0f7624c45aa577af44d8f36c8b4caa"} Oct 30 13:07:12.192950 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:12.192920 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8"] Oct 30 13:07:12.196342 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:12.196319 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0f7d14-7d594659c5-vdlz8"] Oct 30 13:07:12.428284 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:12.428249 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" podUID="110bd521-953f-48c7-990a-823b164a281b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:13.090484 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:13.090449 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69e26174-f25d-4904-b2d8-b69a962ca344" path="/var/lib/kubelet/pods/69e26174-f25d-4904-b2d8-b69a962ca344/volumes" Oct 30 13:07:15.801224 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:15.801181 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" podUID="484b8ef9-46da-48da-a050-97bc7cea4f07" containerName="payload-processing" containerID="cri-o://549e66280e29ec55f73a3cf8acf4704ec48170c7bc9b42a98460d7d2e2074d2b" gracePeriod=25 Oct 30 13:07:15.847931 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:15.847866 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" podUID="29813ba6-6304-4a2f-ab9b-ce9f4d904ad5" containerName="payload-pre-processing" containerID="cri-o://1c5bbc7ad4934a71a5d0ca3f51a3f3bae5fc91c74d80f1468eaab744e706eb8b" gracePeriod=25 Oct 30 13:07:18.137585 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:18.137545 4468 generic.go:356] "Generic (PLEG): container finished" podID="110bd521-953f-48c7-990a-823b164a281b" containerID="22e689d84524c87a0d40c8b99895125f26a3aa09d97442cf9b2900e291ae7dc5" exitCode=0 Oct 30 13:07:18.137984 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:18.137589 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" event={"ID":"110bd521-953f-48c7-990a-823b164a281b","Type":"ContainerDied","Data":"22e689d84524c87a0d40c8b99895125f26a3aa09d97442cf9b2900e291ae7dc5"} Oct 30 13:07:18.174062 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:18.174007 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" Oct 30 13:07:18.329346 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:18.329318 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/110bd521-953f-48c7-990a-823b164a281b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/110bd521-953f-48c7-990a-823b164a281b-plugins-config-volume\") pod \"110bd521-953f-48c7-990a-823b164a281b\" (UID: \"110bd521-953f-48c7-990a-823b164a281b\") " Oct 30 13:07:18.329491 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:18.329479 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/110bd521-953f-48c7-990a-823b164a281b-kube-api-access-d45kt\" (UniqueName: \"kubernetes.io/projected/110bd521-953f-48c7-990a-823b164a281b-kube-api-access-d45kt\") pod \"110bd521-953f-48c7-990a-823b164a281b\" (UID: \"110bd521-953f-48c7-990a-823b164a281b\") " Oct 30 13:07:18.329669 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:18.329648 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110bd521-953f-48c7-990a-823b164a281b-plugins-config-volume" pod "110bd521-953f-48c7-990a-823b164a281b" (UID: "110bd521-953f-48c7-990a-823b164a281b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:18.331517 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:18.331497 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110bd521-953f-48c7-990a-823b164a281b-kube-api-access-d45kt" pod "110bd521-953f-48c7-990a-823b164a281b" (UID: "110bd521-953f-48c7-990a-823b164a281b"). InnerVolumeSpecName "kube-api-access-d45kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:18.430452 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:18.430402 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d45kt\" (UniqueName: \"kubernetes.io/projected/110bd521-953f-48c7-990a-823b164a281b-kube-api-access-d45kt\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:18.430452 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:18.430424 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/110bd521-953f-48c7-990a-823b164a281b-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:18.607891 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:18.607854 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" podUID="484b8ef9-46da-48da-a050-97bc7cea4f07" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:18.608050 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:18.607969 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" podUID="29813ba6-6304-4a2f-ab9b-ce9f4d904ad5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:19.143761 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.143737 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" Oct 30 13:07:19.144141 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.143776 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv" event={"ID":"110bd521-953f-48c7-990a-823b164a281b","Type":"ContainerDied","Data":"9241be5304741dc6c396aded48ebff2b46dc698ad345291e809e23100a1df438"} Oct 30 13:07:19.144141 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.143828 4468 scope.go:122] "RemoveContainer" containerID="22e689d84524c87a0d40c8b99895125f26a3aa09d97442cf9b2900e291ae7dc5" Oct 30 13:07:19.201673 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.201649 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv"] Oct 30 13:07:19.204982 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.204959 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-12fd33-65b8dc7775-l4zkv"] Oct 30 13:07:19.428358 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.428330 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" containerID="cri-o://3590f8f7699f08cafcb0942f859a4e8ad1b113d1a58238fb72ce65edcf0c777d" gracePeriod=45 Oct 30 13:07:19.441581 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.441550 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" containerID="cri-o://df8ab2b5560db3f6e0e1870ee331bf326207a803695a2e9d7c422fc4e2f02fc2" gracePeriod=45 Oct 30 13:07:19.487558 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.487523 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": dial tcp 10.134.0.77:8000: connect: connection refused" Oct 30 13:07:19.725382 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.725360 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 13:07:19.728746 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.728727 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 13:07:19.744593 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.744564 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-home\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-home\") pod \"1903b7f6-f2e3-4000-9399-a7117583640f\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " Oct 30 13:07:19.744766 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.744646 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-tmp-dir\") pod \"fa877b54-11c9-4574-9d5e-e0d00dff0020\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " Oct 30 13:07:19.744766 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.744693 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-model-cache\") pod \"fa877b54-11c9-4574-9d5e-e0d00dff0020\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " Oct 30 13:07:19.744766 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.744730 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa877b54-11c9-4574-9d5e-e0d00dff0020-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa877b54-11c9-4574-9d5e-e0d00dff0020-tls-certs\") pod \"fa877b54-11c9-4574-9d5e-e0d00dff0020\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " Oct 30 13:07:19.744982 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.744788 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-dshm\") pod \"1903b7f6-f2e3-4000-9399-a7117583640f\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " Oct 30 13:07:19.744982 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.744853 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1903b7f6-f2e3-4000-9399-a7117583640f-tls-certs\" (UniqueName: \"kubernetes.io/secret/1903b7f6-f2e3-4000-9399-a7117583640f-tls-certs\") pod \"1903b7f6-f2e3-4000-9399-a7117583640f\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " Oct 30 13:07:19.744982 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.744893 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-dshm\") pod \"fa877b54-11c9-4574-9d5e-e0d00dff0020\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " Oct 30 13:07:19.744982 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.744934 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1903b7f6-f2e3-4000-9399-a7117583640f-kube-api-access-lv4ds\" (UniqueName: \"kubernetes.io/projected/1903b7f6-f2e3-4000-9399-a7117583640f-kube-api-access-lv4ds\") pod \"1903b7f6-f2e3-4000-9399-a7117583640f\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " Oct 30 13:07:19.744982 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.744968 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-home\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-home\") pod \"fa877b54-11c9-4574-9d5e-e0d00dff0020\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " Oct 30 13:07:19.745296 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.745049 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-model-cache\") pod \"1903b7f6-f2e3-4000-9399-a7117583640f\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " Oct 30 13:07:19.745296 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.745163 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa877b54-11c9-4574-9d5e-e0d00dff0020-kube-api-access-kxwjw\" (UniqueName: \"kubernetes.io/projected/fa877b54-11c9-4574-9d5e-e0d00dff0020-kube-api-access-kxwjw\") pod \"fa877b54-11c9-4574-9d5e-e0d00dff0020\" (UID: \"fa877b54-11c9-4574-9d5e-e0d00dff0020\") " Oct 30 13:07:19.745296 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.745224 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-tmp-dir\") pod \"1903b7f6-f2e3-4000-9399-a7117583640f\" (UID: \"1903b7f6-f2e3-4000-9399-a7117583640f\") " Oct 30 13:07:19.745849 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.745811 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-home" pod "fa877b54-11c9-4574-9d5e-e0d00dff0020" (UID: "fa877b54-11c9-4574-9d5e-e0d00dff0020"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.745967 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.745945 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-tmp-dir" pod "1903b7f6-f2e3-4000-9399-a7117583640f" (UID: "1903b7f6-f2e3-4000-9399-a7117583640f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.746387 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.746360 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-model-cache" pod "fa877b54-11c9-4574-9d5e-e0d00dff0020" (UID: "fa877b54-11c9-4574-9d5e-e0d00dff0020"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.746639 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.746617 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-tmp-dir" pod "fa877b54-11c9-4574-9d5e-e0d00dff0020" (UID: "fa877b54-11c9-4574-9d5e-e0d00dff0020"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.746737 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.746718 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-home" pod "1903b7f6-f2e3-4000-9399-a7117583640f" (UID: "1903b7f6-f2e3-4000-9399-a7117583640f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.746806 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.746771 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-model-cache" pod "1903b7f6-f2e3-4000-9399-a7117583640f" (UID: "1903b7f6-f2e3-4000-9399-a7117583640f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.752437 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.752401 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1903b7f6-f2e3-4000-9399-a7117583640f-tls-certs" pod "1903b7f6-f2e3-4000-9399-a7117583640f" (UID: "1903b7f6-f2e3-4000-9399-a7117583640f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:19.752437 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.752412 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-dshm" pod "fa877b54-11c9-4574-9d5e-e0d00dff0020" (UID: "fa877b54-11c9-4574-9d5e-e0d00dff0020"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.752437 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.752424 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-dshm" pod "1903b7f6-f2e3-4000-9399-a7117583640f" (UID: "1903b7f6-f2e3-4000-9399-a7117583640f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.752686 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.752472 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1903b7f6-f2e3-4000-9399-a7117583640f-kube-api-access-lv4ds" pod "1903b7f6-f2e3-4000-9399-a7117583640f" (UID: "1903b7f6-f2e3-4000-9399-a7117583640f"). InnerVolumeSpecName "kube-api-access-lv4ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:19.752686 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.752505 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa877b54-11c9-4574-9d5e-e0d00dff0020-kube-api-access-kxwjw" pod "fa877b54-11c9-4574-9d5e-e0d00dff0020" (UID: "fa877b54-11c9-4574-9d5e-e0d00dff0020"). InnerVolumeSpecName "kube-api-access-kxwjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:19.753775 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.753750 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa877b54-11c9-4574-9d5e-e0d00dff0020-tls-certs" pod "fa877b54-11c9-4574-9d5e-e0d00dff0020" (UID: "fa877b54-11c9-4574-9d5e-e0d00dff0020"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:19.846670 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.846627 4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-home\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.846863 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.846691 4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-tmp-dir\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.846863 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.846709 4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-model-cache\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.846863 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.846723 4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa877b54-11c9-4574-9d5e-e0d00dff0020-tls-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.846863 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.846735 4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-dshm\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.846863 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.846747 4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1903b7f6-f2e3-4000-9399-a7117583640f-tls-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.846863 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.846760 4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-dshm\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.846863 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.846774 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv4ds\" (UniqueName: \"kubernetes.io/projected/1903b7f6-f2e3-4000-9399-a7117583640f-kube-api-access-lv4ds\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.846863 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.846787 4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa877b54-11c9-4574-9d5e-e0d00dff0020-home\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.846863 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.846800 4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-model-cache\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.846863 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.846814 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxwjw\" (UniqueName: \"kubernetes.io/projected/fa877b54-11c9-4574-9d5e-e0d00dff0020-kube-api-access-kxwjw\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.846863 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:19.846826 4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1903b7f6-f2e3-4000-9399-a7117583640f-tmp-dir\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:20.152727 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:20.152638 4468 generic.go:356] "Generic (PLEG): container finished" podID="1903b7f6-f2e3-4000-9399-a7117583640f" containerID="3590f8f7699f08cafcb0942f859a4e8ad1b113d1a58238fb72ce65edcf0c777d" exitCode=0 Oct 30 13:07:20.152727 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:20.152703 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" Oct 30 13:07:20.153295 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:20.152718 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" event={"ID":"1903b7f6-f2e3-4000-9399-a7117583640f","Type":"ContainerDied","Data":"3590f8f7699f08cafcb0942f859a4e8ad1b113d1a58238fb72ce65edcf0c777d"} Oct 30 13:07:20.153295 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:20.152762 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6" event={"ID":"1903b7f6-f2e3-4000-9399-a7117583640f","Type":"ContainerDied","Data":"4d5c7a3bc4589f1331a295d364fd75716b1c9dcae746ecd06a7e1f9389982072"} Oct 30 13:07:20.153295 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:20.152784 4468 scope.go:122] "RemoveContainer" containerID="3590f8f7699f08cafcb0942f859a4e8ad1b113d1a58238fb72ce65edcf0c777d" Oct 30 13:07:20.154774 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:20.154281 4468 generic.go:356] "Generic (PLEG): container finished" podID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerID="df8ab2b5560db3f6e0e1870ee331bf326207a803695a2e9d7c422fc4e2f02fc2" exitCode=0 Oct 30 13:07:20.154774 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:20.154341 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" event={"ID":"fa877b54-11c9-4574-9d5e-e0d00dff0020","Type":"ContainerDied","Data":"df8ab2b5560db3f6e0e1870ee331bf326207a803695a2e9d7c422fc4e2f02fc2"} Oct 30 13:07:20.154774 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:20.154374 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" event={"ID":"fa877b54-11c9-4574-9d5e-e0d00dff0020","Type":"ContainerDied","Data":"811d744ec996fe861bb440980416de3893c267f35e8fc302053893cd1f27dc50"} Oct 30 13:07:20.154774 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:20.154453 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z" Oct 30 13:07:20.166411 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:20.166396 4468 scope.go:122] "RemoveContainer" containerID="3590f8f7699f08cafcb0942f859a4e8ad1b113d1a58238fb72ce65edcf0c777d" Oct 30 13:07:20.166692 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:07:20.166666 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3590f8f7699f08cafcb0942f859a4e8ad1b113d1a58238fb72ce65edcf0c777d\": container with ID starting with 3590f8f7699f08cafcb0942f859a4e8ad1b113d1a58238fb72ce65edcf0c777d not found: ID does not exist" containerID="3590f8f7699f08cafcb0942f859a4e8ad1b113d1a58238fb72ce65edcf0c777d" Oct 30 13:07:20.166814 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:20.166700 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3590f8f7699f08cafcb0942f859a4e8ad1b113d1a58238fb72ce65edcf0c777d"} err="failed to get container status \"3590f8f7699f08cafcb0942f859a4e8ad1b113d1a58238fb72ce65edcf0c777d\": rpc error: code = NotFound desc = could not find container \"3590f8f7699f08cafcb0942f859a4e8ad1b113d1a58238fb72ce65edcf0c777d\": container with ID starting with 3590f8f7699f08cafcb0942f859a4e8ad1b113d1a58238fb72ce65edcf0c777d not found: ID does not exist" Oct 30 13:07:20.166814 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:20.166720 4468 scope.go:122] "RemoveContainer" containerID="df8ab2b5560db3f6e0e1870ee331bf326207a803695a2e9d7c422fc4e2f02fc2" Oct 30 13:07:20.177559 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:20.177524 4468 scope.go:122] "RemoveContainer" containerID="df8ab2b5560db3f6e0e1870ee331bf326207a803695a2e9d7c422fc4e2f02fc2" Oct 30 13:07:20.177853 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:07:20.177831 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8ab2b5560db3f6e0e1870ee331bf326207a803695a2e9d7c422fc4e2f02fc2\": container with ID starting with df8ab2b5560db3f6e0e1870ee331bf326207a803695a2e9d7c422fc4e2f02fc2 not found: ID does not exist" containerID="df8ab2b5560db3f6e0e1870ee331bf326207a803695a2e9d7c422fc4e2f02fc2" Oct 30 13:07:20.177927 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:20.177863 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8ab2b5560db3f6e0e1870ee331bf326207a803695a2e9d7c422fc4e2f02fc2"} err="failed to get container status \"df8ab2b5560db3f6e0e1870ee331bf326207a803695a2e9d7c422fc4e2f02fc2\": rpc error: code = NotFound desc = could not find container \"df8ab2b5560db3f6e0e1870ee331bf326207a803695a2e9d7c422fc4e2f02fc2\": container with ID starting with df8ab2b5560db3f6e0e1870ee331bf326207a803695a2e9d7c422fc4e2f02fc2 not found: ID does not exist" Oct 30 13:07:20.211784 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:20.211758 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6"] Oct 30 13:07:20.217408 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:20.217388 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/distinct-2-w0-f920f7-kserve-7cbcc697b6-82gl6"] Oct 30 13:07:20.265723 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:20.265682 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z"] Oct 30 13:07:20.278445 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:20.278396 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/premium-w0-f920f7-kserve-c85dc7858-h5n2z"] Oct 30 13:07:21.088851 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:21.088816 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="110bd521-953f-48c7-990a-823b164a281b" path="/var/lib/kubelet/pods/110bd521-953f-48c7-990a-823b164a281b/volumes" Oct 30 13:07:21.089377 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:21.089353 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" path="/var/lib/kubelet/pods/1903b7f6-f2e3-4000-9399-a7117583640f/volumes" Oct 30 13:07:21.089823 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:21.089803 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" path="/var/lib/kubelet/pods/fa877b54-11c9-4574-9d5e-e0d00dff0020/volumes" Oct 30 13:07:23.598182 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:07:23.598134 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:07:28.607828 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:28.607788 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" podUID="29813ba6-6304-4a2f-ab9b-ce9f4d904ad5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:28.608330 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:28.607788 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" podUID="484b8ef9-46da-48da-a050-97bc7cea4f07" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:36.125921 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.125895 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" Oct 30 13:07:36.163139 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.163061 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" Oct 30 13:07:36.244526 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.244088 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/484b8ef9-46da-48da-a050-97bc7cea4f07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/484b8ef9-46da-48da-a050-97bc7cea4f07-plugins-config-volume\") pod \"484b8ef9-46da-48da-a050-97bc7cea4f07\" (UID: \"484b8ef9-46da-48da-a050-97bc7cea4f07\") " Oct 30 13:07:36.244526 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.244237 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29813ba6-6304-4a2f-ab9b-ce9f4d904ad5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29813ba6-6304-4a2f-ab9b-ce9f4d904ad5-plugins-config-volume\") pod \"29813ba6-6304-4a2f-ab9b-ce9f4d904ad5\" (UID: \"29813ba6-6304-4a2f-ab9b-ce9f4d904ad5\") " Oct 30 13:07:36.244526 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.244318 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29813ba6-6304-4a2f-ab9b-ce9f4d904ad5-kube-api-access-tq79k\" (UniqueName: \"kubernetes.io/projected/29813ba6-6304-4a2f-ab9b-ce9f4d904ad5-kube-api-access-tq79k\") pod \"29813ba6-6304-4a2f-ab9b-ce9f4d904ad5\" (UID: \"29813ba6-6304-4a2f-ab9b-ce9f4d904ad5\") " Oct 30 13:07:36.244526 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.244378 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/484b8ef9-46da-48da-a050-97bc7cea4f07-kube-api-access-vq54q\" (UniqueName: \"kubernetes.io/projected/484b8ef9-46da-48da-a050-97bc7cea4f07-kube-api-access-vq54q\") pod \"484b8ef9-46da-48da-a050-97bc7cea4f07\" (UID: \"484b8ef9-46da-48da-a050-97bc7cea4f07\") " Oct 30 13:07:36.247194 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.247165 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484b8ef9-46da-48da-a050-97bc7cea4f07-kube-api-access-vq54q" pod "484b8ef9-46da-48da-a050-97bc7cea4f07" (UID: "484b8ef9-46da-48da-a050-97bc7cea4f07"). InnerVolumeSpecName "kube-api-access-vq54q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:36.249329 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.247787 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/484b8ef9-46da-48da-a050-97bc7cea4f07-plugins-config-volume" pod "484b8ef9-46da-48da-a050-97bc7cea4f07" (UID: "484b8ef9-46da-48da-a050-97bc7cea4f07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:36.250415 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.250085 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29813ba6-6304-4a2f-ab9b-ce9f4d904ad5-plugins-config-volume" pod "29813ba6-6304-4a2f-ab9b-ce9f4d904ad5" (UID: "29813ba6-6304-4a2f-ab9b-ce9f4d904ad5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:36.272920 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.272885 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29813ba6-6304-4a2f-ab9b-ce9f4d904ad5-kube-api-access-tq79k" pod "29813ba6-6304-4a2f-ab9b-ce9f4d904ad5" (UID: "29813ba6-6304-4a2f-ab9b-ce9f4d904ad5"). InnerVolumeSpecName "kube-api-access-tq79k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:36.275494 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.275460 4468 generic.go:356] "Generic (PLEG): container finished" podID="29813ba6-6304-4a2f-ab9b-ce9f4d904ad5" containerID="1c5bbc7ad4934a71a5d0ca3f51a3f3bae5fc91c74d80f1468eaab744e706eb8b" exitCode=0 Oct 30 13:07:36.275652 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.275549 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" event={"ID":"29813ba6-6304-4a2f-ab9b-ce9f4d904ad5","Type":"ContainerDied","Data":"1c5bbc7ad4934a71a5d0ca3f51a3f3bae5fc91c74d80f1468eaab744e706eb8b"} Oct 30 13:07:36.275652 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.275600 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" event={"ID":"29813ba6-6304-4a2f-ab9b-ce9f4d904ad5","Type":"ContainerDied","Data":"30ee0e57206074e3934e0a17a95c00fa2048c27676c3d53cfff2b93cd3346db0"} Oct 30 13:07:36.275652 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.275631 4468 scope.go:122] "RemoveContainer" containerID="1c5bbc7ad4934a71a5d0ca3f51a3f3bae5fc91c74d80f1468eaab744e706eb8b" Oct 30 13:07:36.276046 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.275778 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2" Oct 30 13:07:36.287972 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.285294 4468 generic.go:356] "Generic (PLEG): container finished" podID="484b8ef9-46da-48da-a050-97bc7cea4f07" containerID="549e66280e29ec55f73a3cf8acf4704ec48170c7bc9b42a98460d7d2e2074d2b" exitCode=0 Oct 30 13:07:36.287972 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.285509 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" event={"ID":"484b8ef9-46da-48da-a050-97bc7cea4f07","Type":"ContainerDied","Data":"549e66280e29ec55f73a3cf8acf4704ec48170c7bc9b42a98460d7d2e2074d2b"} Oct 30 13:07:36.287972 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.285560 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" event={"ID":"484b8ef9-46da-48da-a050-97bc7cea4f07","Type":"ContainerDied","Data":"3c6ceb6f497ba022fa3a6587103d9579eec3388a75c3d2dce9fc1b3220d04ac7"} Oct 30 13:07:36.287972 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.285625 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4" Oct 30 13:07:36.295658 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.295639 4468 scope.go:122] "RemoveContainer" containerID="1c5bbc7ad4934a71a5d0ca3f51a3f3bae5fc91c74d80f1468eaab744e706eb8b" Oct 30 13:07:36.296002 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:07:36.295970 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c5bbc7ad4934a71a5d0ca3f51a3f3bae5fc91c74d80f1468eaab744e706eb8b\": container with ID starting with 1c5bbc7ad4934a71a5d0ca3f51a3f3bae5fc91c74d80f1468eaab744e706eb8b not found: ID does not exist" containerID="1c5bbc7ad4934a71a5d0ca3f51a3f3bae5fc91c74d80f1468eaab744e706eb8b" Oct 30 13:07:36.296181 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.296000 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5bbc7ad4934a71a5d0ca3f51a3f3bae5fc91c74d80f1468eaab744e706eb8b"} err="failed to get container status \"1c5bbc7ad4934a71a5d0ca3f51a3f3bae5fc91c74d80f1468eaab744e706eb8b\": rpc error: code = NotFound desc = could not find container \"1c5bbc7ad4934a71a5d0ca3f51a3f3bae5fc91c74d80f1468eaab744e706eb8b\": container with ID starting with 1c5bbc7ad4934a71a5d0ca3f51a3f3bae5fc91c74d80f1468eaab744e706eb8b not found: ID does not exist" Oct 30 13:07:36.296181 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.296018 4468 scope.go:122] "RemoveContainer" containerID="549e66280e29ec55f73a3cf8acf4704ec48170c7bc9b42a98460d7d2e2074d2b" Oct 30 13:07:36.304646 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.304589 4468 scope.go:122] "RemoveContainer" containerID="549e66280e29ec55f73a3cf8acf4704ec48170c7bc9b42a98460d7d2e2074d2b" Oct 30 13:07:36.304929 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:07:36.304902 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"549e66280e29ec55f73a3cf8acf4704ec48170c7bc9b42a98460d7d2e2074d2b\": container with ID starting with 549e66280e29ec55f73a3cf8acf4704ec48170c7bc9b42a98460d7d2e2074d2b not found: ID does not exist" containerID="549e66280e29ec55f73a3cf8acf4704ec48170c7bc9b42a98460d7d2e2074d2b" Oct 30 13:07:36.305018 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.304934 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549e66280e29ec55f73a3cf8acf4704ec48170c7bc9b42a98460d7d2e2074d2b"} err="failed to get container status \"549e66280e29ec55f73a3cf8acf4704ec48170c7bc9b42a98460d7d2e2074d2b\": rpc error: code = NotFound desc = could not find container \"549e66280e29ec55f73a3cf8acf4704ec48170c7bc9b42a98460d7d2e2074d2b\": container with ID starting with 549e66280e29ec55f73a3cf8acf4704ec48170c7bc9b42a98460d7d2e2074d2b not found: ID does not exist" Oct 30 13:07:36.334412 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.334350 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2"] Oct 30 13:07:36.337701 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.337683 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f920f7-64c9f64f6c-qs4w2"] Oct 30 13:07:36.346245 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.346224 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq79k\" (UniqueName: \"kubernetes.io/projected/29813ba6-6304-4a2f-ab9b-ce9f4d904ad5-kube-api-access-tq79k\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:36.346329 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.346247 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq54q\" (UniqueName: \"kubernetes.io/projected/484b8ef9-46da-48da-a050-97bc7cea4f07-kube-api-access-vq54q\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:36.346329 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.346257 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/484b8ef9-46da-48da-a050-97bc7cea4f07-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:36.346329 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.346266 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29813ba6-6304-4a2f-ab9b-ce9f4d904ad5-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:36.393263 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.393237 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4"] Oct 30 13:07:36.396699 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:36.396679 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f920f7-6f98685466-qtsf4"] Oct 30 13:07:37.090507 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:37.090420 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29813ba6-6304-4a2f-ab9b-ce9f4d904ad5" path="/var/lib/kubelet/pods/29813ba6-6304-4a2f-ab9b-ce9f4d904ad5/volumes" Oct 30 13:07:37.090908 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:37.090883 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="484b8ef9-46da-48da-a050-97bc7cea4f07" path="/var/lib/kubelet/pods/484b8ef9-46da-48da-a050-97bc7cea4f07/volumes" Oct 30 13:07:42.530174 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:42.530137 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl"] Oct 30 13:07:42.531839 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:42.530379 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" podUID="b7a743c1-ed5e-4fec-aa03-e3fb9f10f048" containerName="maas-api" containerID="cri-o://c5ba133dc68ab2963cc8757ca0f036f22700f307c6e89813169b310defea564c" gracePeriod=30 Oct 30 13:07:42.842886 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:42.842859 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" Oct 30 13:07:43.028752 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.020143 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-kube-api-access-hnpdh\" (UniqueName: \"kubernetes.io/projected/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-kube-api-access-hnpdh\") pod \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\" (UID: \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\") " Oct 30 13:07:43.028752 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.020360 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-openshift-service-ca\") pod \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\" (UID: \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\") " Oct 30 13:07:43.028752 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.020397 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-metrics-certs\") pod \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\" (UID: \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\") " Oct 30 13:07:43.028752 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.020429 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-maas-api-tls\") pod \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\" (UID: \"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048\") " Oct 30 13:07:43.029767 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.029726 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-openshift-service-ca" pod "b7a743c1-ed5e-4fec-aa03-e3fb9f10f048" (UID: "b7a743c1-ed5e-4fec-aa03-e3fb9f10f048"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:43.032519 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.032491 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-metrics-certs" pod "b7a743c1-ed5e-4fec-aa03-e3fb9f10f048" (UID: "b7a743c1-ed5e-4fec-aa03-e3fb9f10f048"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:43.033114 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.033049 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-maas-api-tls" pod "b7a743c1-ed5e-4fec-aa03-e3fb9f10f048" (UID: "b7a743c1-ed5e-4fec-aa03-e3fb9f10f048"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:43.053905 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.053844 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-kube-api-access-hnpdh" pod "b7a743c1-ed5e-4fec-aa03-e3fb9f10f048" (UID: "b7a743c1-ed5e-4fec-aa03-e3fb9f10f048"). InnerVolumeSpecName "kube-api-access-hnpdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:43.122452 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.122422 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:43.122452 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.122448 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:43.122452 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.122458 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:43.122653 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.122467 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnpdh\" (UniqueName: \"kubernetes.io/projected/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048-kube-api-access-hnpdh\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:43.341556 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.341514 4468 generic.go:356] "Generic (PLEG): container finished" podID="b7a743c1-ed5e-4fec-aa03-e3fb9f10f048" containerID="c5ba133dc68ab2963cc8757ca0f036f22700f307c6e89813169b310defea564c" exitCode=0 Oct 30 13:07:43.341735 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.341594 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" Oct 30 13:07:43.341735 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.341604 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" event={"ID":"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048","Type":"ContainerDied","Data":"c5ba133dc68ab2963cc8757ca0f036f22700f307c6e89813169b310defea564c"} Oct 30 13:07:43.341735 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.341665 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl" event={"ID":"b7a743c1-ed5e-4fec-aa03-e3fb9f10f048","Type":"ContainerDied","Data":"62c4e48519a3ae04bdad9f5d8cba47042176593168f696973b796d91954aa835"} Oct 30 13:07:43.341735 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.341688 4468 scope.go:122] "RemoveContainer" containerID="c5ba133dc68ab2963cc8757ca0f036f22700f307c6e89813169b310defea564c" Oct 30 13:07:43.350937 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.350919 4468 scope.go:122] "RemoveContainer" containerID="c5ba133dc68ab2963cc8757ca0f036f22700f307c6e89813169b310defea564c" Oct 30 13:07:43.351237 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:07:43.351216 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ba133dc68ab2963cc8757ca0f036f22700f307c6e89813169b310defea564c\": container with ID starting with c5ba133dc68ab2963cc8757ca0f036f22700f307c6e89813169b310defea564c not found: ID does not exist" containerID="c5ba133dc68ab2963cc8757ca0f036f22700f307c6e89813169b310defea564c" Oct 30 13:07:43.351298 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.351246 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ba133dc68ab2963cc8757ca0f036f22700f307c6e89813169b310defea564c"} err="failed to get container status \"c5ba133dc68ab2963cc8757ca0f036f22700f307c6e89813169b310defea564c\": rpc error: code = NotFound desc = could not find container \"c5ba133dc68ab2963cc8757ca0f036f22700f307c6e89813169b310defea564c\": container with ID starting with c5ba133dc68ab2963cc8757ca0f036f22700f307c6e89813169b310defea564c not found: ID does not exist" Oct 30 13:07:43.453320 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.453296 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl"] Oct 30 13:07:43.485784 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:43.485759 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f920f7-6d79468588-bl5nl"] Oct 30 13:07:45.089129 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:45.089093 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7a743c1-ed5e-4fec-aa03-e3fb9f10f048" path="/var/lib/kubelet/pods/b7a743c1-ed5e-4fec-aa03-e3fb9f10f048/volumes" Oct 30 13:07:45.314627 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:45.314592 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx"] Oct 30 13:07:46.986954 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:46.986919 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm"] Oct 30 13:07:48.501788 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:48.501747 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5"] Oct 30 13:07:48.502280 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:48.502025 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" podUID="325f91fe-140a-47b9-8b30-c66476278515" containerName="maas-api" containerID="cri-o://198a4540276fb158ebf1b9b9a1a8e2249924d82ce08dfe5ebc4db0e2f05d7a32" gracePeriod=30 Oct 30 13:07:48.782727 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:48.782701 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" Oct 30 13:07:48.881237 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:48.881205 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/325f91fe-140a-47b9-8b30-c66476278515-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/325f91fe-140a-47b9-8b30-c66476278515-maas-api-tls\") pod \"325f91fe-140a-47b9-8b30-c66476278515\" (UID: \"325f91fe-140a-47b9-8b30-c66476278515\") " Oct 30 13:07:48.881437 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:48.881321 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/325f91fe-140a-47b9-8b30-c66476278515-metrics-certs\" (UniqueName: \"kubernetes.io/secret/325f91fe-140a-47b9-8b30-c66476278515-metrics-certs\") pod \"325f91fe-140a-47b9-8b30-c66476278515\" (UID: \"325f91fe-140a-47b9-8b30-c66476278515\") " Oct 30 13:07:48.881437 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:48.881381 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/325f91fe-140a-47b9-8b30-c66476278515-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/325f91fe-140a-47b9-8b30-c66476278515-openshift-service-ca\") pod \"325f91fe-140a-47b9-8b30-c66476278515\" (UID: \"325f91fe-140a-47b9-8b30-c66476278515\") " Oct 30 13:07:48.881557 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:48.881446 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/325f91fe-140a-47b9-8b30-c66476278515-kube-api-access-rrzgk\" (UniqueName: \"kubernetes.io/projected/325f91fe-140a-47b9-8b30-c66476278515-kube-api-access-rrzgk\") pod \"325f91fe-140a-47b9-8b30-c66476278515\" (UID: \"325f91fe-140a-47b9-8b30-c66476278515\") " Oct 30 13:07:48.881849 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:48.881822 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/325f91fe-140a-47b9-8b30-c66476278515-openshift-service-ca" pod "325f91fe-140a-47b9-8b30-c66476278515" (UID: "325f91fe-140a-47b9-8b30-c66476278515"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:48.883457 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:48.883434 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/325f91fe-140a-47b9-8b30-c66476278515-maas-api-tls" pod "325f91fe-140a-47b9-8b30-c66476278515" (UID: "325f91fe-140a-47b9-8b30-c66476278515"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:48.883561 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:48.883489 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/325f91fe-140a-47b9-8b30-c66476278515-metrics-certs" pod "325f91fe-140a-47b9-8b30-c66476278515" (UID: "325f91fe-140a-47b9-8b30-c66476278515"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:48.883626 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:48.883600 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325f91fe-140a-47b9-8b30-c66476278515-kube-api-access-rrzgk" pod "325f91fe-140a-47b9-8b30-c66476278515" (UID: "325f91fe-140a-47b9-8b30-c66476278515"). InnerVolumeSpecName "kube-api-access-rrzgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:48.985684 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:48.985233 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/325f91fe-140a-47b9-8b30-c66476278515-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:48.985684 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:48.985270 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/325f91fe-140a-47b9-8b30-c66476278515-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:48.985684 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:48.985285 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/325f91fe-140a-47b9-8b30-c66476278515-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:48.985684 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:48.985300 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrzgk\" (UniqueName: \"kubernetes.io/projected/325f91fe-140a-47b9-8b30-c66476278515-kube-api-access-rrzgk\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:49.388563 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:49.388478 4468 generic.go:356] "Generic (PLEG): container finished" podID="325f91fe-140a-47b9-8b30-c66476278515" containerID="198a4540276fb158ebf1b9b9a1a8e2249924d82ce08dfe5ebc4db0e2f05d7a32" exitCode=0 Oct 30 13:07:49.388563 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:49.388525 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" Oct 30 13:07:49.389262 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:49.388562 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" event={"ID":"325f91fe-140a-47b9-8b30-c66476278515","Type":"ContainerDied","Data":"198a4540276fb158ebf1b9b9a1a8e2249924d82ce08dfe5ebc4db0e2f05d7a32"} Oct 30 13:07:49.389262 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:49.388622 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5" event={"ID":"325f91fe-140a-47b9-8b30-c66476278515","Type":"ContainerDied","Data":"4fc6b1d879430fd9d7a22e0e00267d363194ed33da5e817e186c51a00c797a5d"} Oct 30 13:07:49.389262 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:49.388645 4468 scope.go:122] "RemoveContainer" containerID="198a4540276fb158ebf1b9b9a1a8e2249924d82ce08dfe5ebc4db0e2f05d7a32" Oct 30 13:07:49.398054 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:49.398035 4468 scope.go:122] "RemoveContainer" containerID="198a4540276fb158ebf1b9b9a1a8e2249924d82ce08dfe5ebc4db0e2f05d7a32" Oct 30 13:07:49.398371 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:07:49.398335 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"198a4540276fb158ebf1b9b9a1a8e2249924d82ce08dfe5ebc4db0e2f05d7a32\": container with ID starting with 198a4540276fb158ebf1b9b9a1a8e2249924d82ce08dfe5ebc4db0e2f05d7a32 not found: ID does not exist" containerID="198a4540276fb158ebf1b9b9a1a8e2249924d82ce08dfe5ebc4db0e2f05d7a32" Oct 30 13:07:49.398464 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:49.398376 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198a4540276fb158ebf1b9b9a1a8e2249924d82ce08dfe5ebc4db0e2f05d7a32"} err="failed to get container status \"198a4540276fb158ebf1b9b9a1a8e2249924d82ce08dfe5ebc4db0e2f05d7a32\": rpc error: code = NotFound desc = could not find container \"198a4540276fb158ebf1b9b9a1a8e2249924d82ce08dfe5ebc4db0e2f05d7a32\": container with ID starting with 198a4540276fb158ebf1b9b9a1a8e2249924d82ce08dfe5ebc4db0e2f05d7a32 not found: ID does not exist" Oct 30 13:07:49.456902 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:49.456876 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5"] Oct 30 13:07:49.472340 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:49.472316 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fddaa3-8775567f7-gnqq5"] Oct 30 13:07:50.315153 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:50.315114 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" podUID="e8e11991-76bb-4c45-8393-33dbc1ed877f" containerName="payload-processing" containerID="cri-o://dbcd4565c7d3b5508901ef4d453230a3daea9e259c786a25456848fab54fcf2c" gracePeriod=25 Oct 30 13:07:51.089722 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:51.089688 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="325f91fe-140a-47b9-8b30-c66476278515" path="/var/lib/kubelet/pods/325f91fe-140a-47b9-8b30-c66476278515/volumes" Oct 30 13:07:51.987495 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:51.987432 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" podUID="35ef76d8-4829-4123-bbff-e7b36c59afba" containerName="payload-pre-processing" containerID="cri-o://6b96d53841725417ee5ab9a88eaf75c34a158c87944593ff48235f3d51639829" gracePeriod=25 Oct 30 13:07:53.598706 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:07:53.598671 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:07:53.680330 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:53.680293 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" podUID="e8e11991-76bb-4c45-8393-33dbc1ed877f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:58.643160 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:58.643122 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" podUID="35ef76d8-4829-4123-bbff-e7b36c59afba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:59.601120 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.601085 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv"] Oct 30 13:07:59.601311 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.601290 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" podUID="5929f380-6cc0-4ccf-97b8-a3e54b01430f" containerName="discovery" containerID="cri-o://7c8414bc728f0a4b4b4aaf40400c1f9d5bc1d1fbc11362cff6d2f3f6e2e83abe" gracePeriod=30 Oct 30 13:07:59.855395 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.855330 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 13:07:59.896106 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.896059 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-token\" (UniqueName: \"kubernetes.io/projected/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-token\") pod \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " Oct 30 13:07:59.896253 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.896145 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-kubeconfig\") pod \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " Oct 30 13:07:59.896253 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.896229 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5929f380-6cc0-4ccf-97b8-a3e54b01430f-kube-api-access-7rwvd\" (UniqueName: \"kubernetes.io/projected/5929f380-6cc0-4ccf-97b8-a3e54b01430f-kube-api-access-7rwvd\") pod \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " Oct 30 13:07:59.896382 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.896269 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-csr-dns-cert\") pod \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " Oct 30 13:07:59.896382 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.896339 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5929f380-6cc0-4ccf-97b8-a3e54b01430f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5929f380-6cc0-4ccf-97b8-a3e54b01430f-local-certs\") pod \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " Oct 30 13:07:59.896382 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.896372 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-cacerts\" (UniqueName: \"kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-cacerts\") pod \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " Oct 30 13:07:59.896552 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.896408 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-csr-ca-configmap\") pod \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\" (UID: \"5929f380-6cc0-4ccf-97b8-a3e54b01430f\") " Oct 30 13:07:59.896925 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.896894 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-csr-ca-configmap" pod "5929f380-6cc0-4ccf-97b8-a3e54b01430f" (UID: "5929f380-6cc0-4ccf-97b8-a3e54b01430f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:59.897080 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.897002 4468 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-csr-ca-configmap\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:59.899379 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.899337 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-token" pod "5929f380-6cc0-4ccf-97b8-a3e54b01430f" (UID: "5929f380-6cc0-4ccf-97b8-a3e54b01430f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:59.899499 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.899440 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-csr-dns-cert" pod "5929f380-6cc0-4ccf-97b8-a3e54b01430f" (UID: "5929f380-6cc0-4ccf-97b8-a3e54b01430f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:59.899671 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.899646 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5929f380-6cc0-4ccf-97b8-a3e54b01430f-local-certs" pod "5929f380-6cc0-4ccf-97b8-a3e54b01430f" (UID: "5929f380-6cc0-4ccf-97b8-a3e54b01430f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:59.899764 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.899648 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-cacerts" pod "5929f380-6cc0-4ccf-97b8-a3e54b01430f" (UID: "5929f380-6cc0-4ccf-97b8-a3e54b01430f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:59.899938 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.899918 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-kubeconfig" pod "5929f380-6cc0-4ccf-97b8-a3e54b01430f" (UID: "5929f380-6cc0-4ccf-97b8-a3e54b01430f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:59.900343 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.900304 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5929f380-6cc0-4ccf-97b8-a3e54b01430f-kube-api-access-7rwvd" pod "5929f380-6cc0-4ccf-97b8-a3e54b01430f" (UID: "5929f380-6cc0-4ccf-97b8-a3e54b01430f"). InnerVolumeSpecName "kube-api-access-7rwvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:59.998601 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.998568 4468 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5929f380-6cc0-4ccf-97b8-a3e54b01430f-local-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:59.998601 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.998596 4468 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-cacerts\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:59.998601 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.998604 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:59.998837 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.998612 4468 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-kubeconfig\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:59.998837 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.998621 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rwvd\" (UniqueName: \"kubernetes.io/projected/5929f380-6cc0-4ccf-97b8-a3e54b01430f-kube-api-access-7rwvd\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:07:59.998837 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:07:59.998630 4468 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5929f380-6cc0-4ccf-97b8-a3e54b01430f-istio-csr-dns-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:00.293226 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.293192 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb"] Oct 30 13:08:00.293425 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.293408 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" podUID="9bf2843b-7deb-4ffa-8b55-48e59ad1f756" containerName="maas-api" containerID="cri-o://1e560430c484a1cdc67aba5febed6504d78abe85beb4286c2783bc7cd9d87063" gracePeriod=30 Oct 30 13:08:00.461691 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.461660 4468 generic.go:356] "Generic (PLEG): container finished" podID="5929f380-6cc0-4ccf-97b8-a3e54b01430f" containerID="7c8414bc728f0a4b4b4aaf40400c1f9d5bc1d1fbc11362cff6d2f3f6e2e83abe" exitCode=0 Oct 30 13:08:00.461873 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.461707 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" Oct 30 13:08:00.461873 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.461742 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" event={"ID":"5929f380-6cc0-4ccf-97b8-a3e54b01430f","Type":"ContainerDied","Data":"7c8414bc728f0a4b4b4aaf40400c1f9d5bc1d1fbc11362cff6d2f3f6e2e83abe"} Oct 30 13:08:00.461873 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.461802 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv" event={"ID":"5929f380-6cc0-4ccf-97b8-a3e54b01430f","Type":"ContainerDied","Data":"1ecd97670f57c65bdc2c05c7aa753235466d63f3366a5ca36e68caa0bb0b3ebc"} Oct 30 13:08:00.461873 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.461827 4468 scope.go:122] "RemoveContainer" containerID="7c8414bc728f0a4b4b4aaf40400c1f9d5bc1d1fbc11362cff6d2f3f6e2e83abe" Oct 30 13:08:00.463564 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.463536 4468 generic.go:356] "Generic (PLEG): container finished" podID="9bf2843b-7deb-4ffa-8b55-48e59ad1f756" containerID="1e560430c484a1cdc67aba5febed6504d78abe85beb4286c2783bc7cd9d87063" exitCode=0 Oct 30 13:08:00.463698 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.463595 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" event={"ID":"9bf2843b-7deb-4ffa-8b55-48e59ad1f756","Type":"ContainerDied","Data":"1e560430c484a1cdc67aba5febed6504d78abe85beb4286c2783bc7cd9d87063"} Oct 30 13:08:00.486968 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.486948 4468 scope.go:122] "RemoveContainer" containerID="7c8414bc728f0a4b4b4aaf40400c1f9d5bc1d1fbc11362cff6d2f3f6e2e83abe" Oct 30 13:08:00.487262 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:08:00.487234 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8414bc728f0a4b4b4aaf40400c1f9d5bc1d1fbc11362cff6d2f3f6e2e83abe\": container with ID starting with 7c8414bc728f0a4b4b4aaf40400c1f9d5bc1d1fbc11362cff6d2f3f6e2e83abe not found: ID does not exist" containerID="7c8414bc728f0a4b4b4aaf40400c1f9d5bc1d1fbc11362cff6d2f3f6e2e83abe" Oct 30 13:08:00.487385 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.487271 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8414bc728f0a4b4b4aaf40400c1f9d5bc1d1fbc11362cff6d2f3f6e2e83abe"} err="failed to get container status \"7c8414bc728f0a4b4b4aaf40400c1f9d5bc1d1fbc11362cff6d2f3f6e2e83abe\": rpc error: code = NotFound desc = could not find container \"7c8414bc728f0a4b4b4aaf40400c1f9d5bc1d1fbc11362cff6d2f3f6e2e83abe\": container with ID starting with 7c8414bc728f0a4b4b4aaf40400c1f9d5bc1d1fbc11362cff6d2f3f6e2e83abe not found: ID does not exist" Oct 30 13:08:00.530665 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.530638 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv"] Oct 30 13:08:00.540165 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.540115 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" Oct 30 13:08:00.541788 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.541763 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h89vv"] Oct 30 13:08:00.605505 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.605431 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-maas-api-tls\") pod \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\" (UID: \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\") " Oct 30 13:08:00.605641 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.605528 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-kube-api-access-8pq9r\" (UniqueName: \"kubernetes.io/projected/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-kube-api-access-8pq9r\") pod \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\" (UID: \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\") " Oct 30 13:08:00.605641 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.605601 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-metrics-certs\") pod \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\" (UID: \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\") " Oct 30 13:08:00.605774 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.605687 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-openshift-service-ca\") pod \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\" (UID: \"9bf2843b-7deb-4ffa-8b55-48e59ad1f756\") " Oct 30 13:08:00.606141 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.606110 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-openshift-service-ca" pod "9bf2843b-7deb-4ffa-8b55-48e59ad1f756" (UID: "9bf2843b-7deb-4ffa-8b55-48e59ad1f756"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:00.607755 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.607729 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-maas-api-tls" pod "9bf2843b-7deb-4ffa-8b55-48e59ad1f756" (UID: "9bf2843b-7deb-4ffa-8b55-48e59ad1f756"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:00.607854 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.607770 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-metrics-certs" pod "9bf2843b-7deb-4ffa-8b55-48e59ad1f756" (UID: "9bf2843b-7deb-4ffa-8b55-48e59ad1f756"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:00.608053 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.608028 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-kube-api-access-8pq9r" pod "9bf2843b-7deb-4ffa-8b55-48e59ad1f756" (UID: "9bf2843b-7deb-4ffa-8b55-48e59ad1f756"). InnerVolumeSpecName "kube-api-access-8pq9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:00.707727 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.707690 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pq9r\" (UniqueName: \"kubernetes.io/projected/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-kube-api-access-8pq9r\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:00.707872 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.707730 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:00.707872 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.707745 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:00.707872 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:00.707757 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf2843b-7deb-4ffa-8b55-48e59ad1f756-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:01.089018 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:01.088990 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5929f380-6cc0-4ccf-97b8-a3e54b01430f" path="/var/lib/kubelet/pods/5929f380-6cc0-4ccf-97b8-a3e54b01430f/volumes" Oct 30 13:08:01.471218 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:01.471186 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" event={"ID":"9bf2843b-7deb-4ffa-8b55-48e59ad1f756","Type":"ContainerDied","Data":"ebf9c241a82f12a5dba81eef5e9e9a31a6921a2f1235cfe3e695d151b25dfe0c"} Oct 30 13:08:01.471218 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:01.471204 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb" Oct 30 13:08:01.471484 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:01.471253 4468 scope.go:122] "RemoveContainer" containerID="1e560430c484a1cdc67aba5febed6504d78abe85beb4286c2783bc7cd9d87063" Oct 30 13:08:01.531266 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:01.531236 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb"] Oct 30 13:08:01.544325 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:01.544300 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9d45fb-b569db4cc-v99zb"] Oct 30 13:08:03.088349 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:03.088314 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf2843b-7deb-4ffa-8b55-48e59ad1f756" path="/var/lib/kubelet/pods/9bf2843b-7deb-4ffa-8b55-48e59ad1f756/volumes" Oct 30 13:08:03.680453 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:03.680420 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" podUID="e8e11991-76bb-4c45-8393-33dbc1ed877f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:05.751610 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:05.751574 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq"] Oct 30 13:08:08.643023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:08.642989 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" podUID="35ef76d8-4829-4123-bbff-e7b36c59afba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:09.276004 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:09.275966 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r"] Oct 30 13:08:09.276263 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:09.276235 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" podUID="b77483d1-fb3c-4be6-8d49-c6863bfcc84d" containerName="istio-proxy" containerID="cri-o://4034c93f561b1ef45372c69924d5859fc11ad2be3b45f227d1eed9c1f21e5184" gracePeriod=30 Oct 30 13:08:10.528382 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:10.528351 4468 generic.go:356] "Generic (PLEG): container finished" podID="e8e11991-76bb-4c45-8393-33dbc1ed877f" containerID="dbcd4565c7d3b5508901ef4d453230a3daea9e259c786a25456848fab54fcf2c" exitCode=0 Oct 30 13:08:10.528803 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:10.528427 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" event={"ID":"e8e11991-76bb-4c45-8393-33dbc1ed877f","Type":"ContainerDied","Data":"dbcd4565c7d3b5508901ef4d453230a3daea9e259c786a25456848fab54fcf2c"} Oct 30 13:08:10.585548 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:10.585495 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" Oct 30 13:08:10.711852 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:10.711819 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e11991-76bb-4c45-8393-33dbc1ed877f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e11991-76bb-4c45-8393-33dbc1ed877f-plugins-config-volume\") pod \"e8e11991-76bb-4c45-8393-33dbc1ed877f\" (UID: \"e8e11991-76bb-4c45-8393-33dbc1ed877f\") " Oct 30 13:08:10.712001 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:10.711906 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e11991-76bb-4c45-8393-33dbc1ed877f-kube-api-access-pdzz8\" (UniqueName: \"kubernetes.io/projected/e8e11991-76bb-4c45-8393-33dbc1ed877f-kube-api-access-pdzz8\") pod \"e8e11991-76bb-4c45-8393-33dbc1ed877f\" (UID: \"e8e11991-76bb-4c45-8393-33dbc1ed877f\") " Oct 30 13:08:10.712216 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:10.712193 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e11991-76bb-4c45-8393-33dbc1ed877f-plugins-config-volume" pod "e8e11991-76bb-4c45-8393-33dbc1ed877f" (UID: "e8e11991-76bb-4c45-8393-33dbc1ed877f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:10.713917 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:10.713888 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e11991-76bb-4c45-8393-33dbc1ed877f-kube-api-access-pdzz8" pod "e8e11991-76bb-4c45-8393-33dbc1ed877f" (UID: "e8e11991-76bb-4c45-8393-33dbc1ed877f"). InnerVolumeSpecName "kube-api-access-pdzz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:10.812928 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:10.812897 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e11991-76bb-4c45-8393-33dbc1ed877f-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:10.812928 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:10.812926 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdzz8\" (UniqueName: \"kubernetes.io/projected/e8e11991-76bb-4c45-8393-33dbc1ed877f-kube-api-access-pdzz8\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:11.535772 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.535745 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" Oct 30 13:08:11.536228 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.535738 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx" event={"ID":"e8e11991-76bb-4c45-8393-33dbc1ed877f","Type":"ContainerDied","Data":"669a8bb959cfe1f4b46d9c8b7cc95cdf255ae90784e026d6f691ce314a3bf359"} Oct 30 13:08:11.536228 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.535875 4468 scope.go:122] "RemoveContainer" containerID="dbcd4565c7d3b5508901ef4d453230a3daea9e259c786a25456848fab54fcf2c" Oct 30 13:08:11.571590 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.571560 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p"] Oct 30 13:08:11.571955 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.571927 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" podUID="4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd" containerName="maas-api" containerID="cri-o://06ab4a4bd6c138d58f54bba8f63c4d20a8942f2e25f2b94bb94c24bb79eb606e" gracePeriod=30 Oct 30 13:08:11.603043 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.603011 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx"] Oct 30 13:08:11.606524 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.606496 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-42220c-855c748588-jwrnx"] Oct 30 13:08:11.816144 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.812526 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn"] Oct 30 13:08:11.844342 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.844313 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s"] Oct 30 13:08:11.855707 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.855685 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" Oct 30 13:08:11.992113 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992064 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf"] Oct 30 13:08:11.992767 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992739 4468 cpu_manager.go:414] "containerMap: removing container" podUID="ee548a01-d0b9-4387-a699-041ac6e9b205" containerName="cleanup" Oct 30 13:08:11.992767 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992765 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee548a01-d0b9-4387-a699-041ac6e9b205" containerName="cleanup" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992787 4468 cpu_manager.go:414] "containerMap: removing container" podUID="af770c09-3be0-418b-8737-3cd556b08035" containerName="maas-api" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992796 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af770c09-3be0-418b-8737-3cd556b08035" containerName="maas-api" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992811 4468 cpu_manager.go:414] "containerMap: removing container" podUID="0b15b44d-dfe6-4001-966b-f72c56f449c1" containerName="istio-proxy" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992819 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b15b44d-dfe6-4001-966b-f72c56f449c1" containerName="istio-proxy" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992834 4468 cpu_manager.go:414] "containerMap: removing container" podUID="36dd06a2-cbb4-43df-8411-f3b35e06e98c" containerName="payload-processing" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992842 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36dd06a2-cbb4-43df-8411-f3b35e06e98c" containerName="payload-processing" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992851 4468 cpu_manager.go:414] "containerMap: removing container" podUID="de938051-680d-48df-80b7-8e71ee81217b" containerName="cleanup" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992858 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de938051-680d-48df-80b7-8e71ee81217b" containerName="cleanup" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992870 4468 cpu_manager.go:414] "containerMap: removing container" podUID="e8e11991-76bb-4c45-8393-33dbc1ed877f" containerName="payload-processing" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992884 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e11991-76bb-4c45-8393-33dbc1ed877f" containerName="payload-processing" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992895 4468 cpu_manager.go:414] "containerMap: removing container" podUID="0b15b44d-dfe6-4001-966b-f72c56f449c1" containerName="istio-proxy" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992902 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b15b44d-dfe6-4001-966b-f72c56f449c1" containerName="istio-proxy" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992911 4468 cpu_manager.go:414] "containerMap: removing container" podUID="416eb2ba-b5b4-4e25-a057-d870df27c1f7" containerName="payload-processing" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992918 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416eb2ba-b5b4-4e25-a057-d870df27c1f7" containerName="payload-processing" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992929 4468 cpu_manager.go:414] "containerMap: removing container" podUID="9f68b80b-1983-4fc9-907f-20653341e98a" containerName="payload-pre-processing" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992936 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f68b80b-1983-4fc9-907f-20653341e98a" containerName="payload-pre-processing" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992952 4468 cpu_manager.go:414] "containerMap: removing container" podUID="29813ba6-6304-4a2f-ab9b-ce9f4d904ad5" containerName="payload-pre-processing" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992960 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29813ba6-6304-4a2f-ab9b-ce9f4d904ad5" containerName="payload-pre-processing" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992973 4468 cpu_manager.go:414] "containerMap: removing container" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992980 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992989 4468 cpu_manager.go:414] "containerMap: removing container" podUID="64964ff4-80ce-4c89-af7b-19d3b24b07f4" containerName="payload-processing" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.992995 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64964ff4-80ce-4c89-af7b-19d3b24b07f4" containerName="payload-processing" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993007 4468 cpu_manager.go:414] "containerMap: removing container" podUID="db33d91a-07fb-4c52-832e-0440bc4654af" containerName="istio-proxy" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993014 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db33d91a-07fb-4c52-832e-0440bc4654af" containerName="istio-proxy" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993023 4468 cpu_manager.go:414] "containerMap: removing container" podUID="065b66a4-c80e-4dde-8bf6-4f14386e6d87" containerName="payload-pre-processing" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993030 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="065b66a4-c80e-4dde-8bf6-4f14386e6d87" containerName="payload-pre-processing" Oct 30 13:08:11.993023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993040 4468 cpu_manager.go:414] "containerMap: removing container" podUID="0f9ddb0a-51f8-4d38-9605-74dc25689e46" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993047 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9ddb0a-51f8-4d38-9605-74dc25689e46" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993060 4468 cpu_manager.go:414] "containerMap: removing container" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993079 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993090 4468 cpu_manager.go:414] "containerMap: removing container" podUID="6bac85e8-7e06-4933-a757-4739e81a0afd" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993097 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bac85e8-7e06-4933-a757-4739e81a0afd" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993107 4468 cpu_manager.go:414] "containerMap: removing container" podUID="bc9a08f6-1257-4e45-9a7d-9687d6554377" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993114 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc9a08f6-1257-4e45-9a7d-9687d6554377" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993124 4468 cpu_manager.go:414] "containerMap: removing container" podUID="16dae47c-220c-4d83-89e5-52d1746775e8" containerName="istio-proxy" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993131 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16dae47c-220c-4d83-89e5-52d1746775e8" containerName="istio-proxy" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993141 4468 cpu_manager.go:414] "containerMap: removing container" podUID="b60e79ec-6b0e-4725-81f3-d02c72fc8efb" containerName="istio-proxy" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993148 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b60e79ec-6b0e-4725-81f3-d02c72fc8efb" containerName="istio-proxy" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993158 4468 cpu_manager.go:414] "containerMap: removing container" podUID="bcf35b28-b756-4281-80d7-bc361b2d56e3" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993165 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf35b28-b756-4281-80d7-bc361b2d56e3" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993175 4468 cpu_manager.go:414] "containerMap: removing container" podUID="6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993180 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993186 4468 cpu_manager.go:414] "containerMap: removing container" podUID="9eb0a51c-3364-487b-8aec-77c2cb215dd7" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993190 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb0a51c-3364-487b-8aec-77c2cb215dd7" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993197 4468 cpu_manager.go:414] "containerMap: removing container" podUID="e977ddbb-e36d-4056-b775-d91dbbb91881" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993201 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e977ddbb-e36d-4056-b775-d91dbbb91881" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993207 4468 cpu_manager.go:414] "containerMap: removing container" podUID="557e9c97-61e1-4358-8249-c79d946c5a9c" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993211 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="557e9c97-61e1-4358-8249-c79d946c5a9c" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993217 4468 cpu_manager.go:414] "containerMap: removing container" podUID="de938051-680d-48df-80b7-8e71ee81217b" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993222 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de938051-680d-48df-80b7-8e71ee81217b" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993232 4468 cpu_manager.go:414] "containerMap: removing container" podUID="9bf2843b-7deb-4ffa-8b55-48e59ad1f756" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993236 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf2843b-7deb-4ffa-8b55-48e59ad1f756" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993242 4468 cpu_manager.go:414] "containerMap: removing container" podUID="6e29344f-41fb-4929-aee9-739cf692fa68" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993246 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e29344f-41fb-4929-aee9-739cf692fa68" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993251 4468 cpu_manager.go:414] "containerMap: removing container" podUID="137214d6-09d8-4a4f-bf37-fcef28d37d07" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993255 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137214d6-09d8-4a4f-bf37-fcef28d37d07" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993262 4468 cpu_manager.go:414] "containerMap: removing container" podUID="2b10586a-875c-4812-9bd2-0e65b81d0224" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993267 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b10586a-875c-4812-9bd2-0e65b81d0224" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993275 4468 cpu_manager.go:414] "containerMap: removing container" podUID="5ae0dc66-24bc-4aad-a275-bc7a8a2c3206" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993279 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae0dc66-24bc-4aad-a275-bc7a8a2c3206" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993286 4468 cpu_manager.go:414] "containerMap: removing container" podUID="325f91fe-140a-47b9-8b30-c66476278515" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993290 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="325f91fe-140a-47b9-8b30-c66476278515" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993299 4468 cpu_manager.go:414] "containerMap: removing container" podUID="1270203a-f11c-4b92-9cb6-b76ab1bbc2a4" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993304 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1270203a-f11c-4b92-9cb6-b76ab1bbc2a4" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993310 4468 cpu_manager.go:414] "containerMap: removing container" podUID="54a87f6f-513a-4a7b-a904-be6b4f218570" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993315 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a87f6f-513a-4a7b-a904-be6b4f218570" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993322 4468 cpu_manager.go:414] "containerMap: removing container" podUID="57f76bd7-cafb-48ad-9f99-8a7b88538a85" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993326 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f76bd7-cafb-48ad-9f99-8a7b88538a85" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993333 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3314d4c4-7905-42dd-b342-277da538ba35" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993337 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3314d4c4-7905-42dd-b342-277da538ba35" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993343 4468 cpu_manager.go:414] "containerMap: removing container" podUID="5929f380-6cc0-4ccf-97b8-a3e54b01430f" containerName="discovery" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993347 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5929f380-6cc0-4ccf-97b8-a3e54b01430f" containerName="discovery" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993354 4468 cpu_manager.go:414] "containerMap: removing container" podUID="fca7e7d8-d36f-4378-8052-04a62530fff6" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993358 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca7e7d8-d36f-4378-8052-04a62530fff6" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993364 4468 cpu_manager.go:414] "containerMap: removing container" podUID="110bd521-953f-48c7-990a-823b164a281b" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993369 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="110bd521-953f-48c7-990a-823b164a281b" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993374 4468 cpu_manager.go:414] "containerMap: removing container" podUID="cd5da4da-cc71-4a46-a697-396a96373776" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993378 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd5da4da-cc71-4a46-a697-396a96373776" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993383 4468 cpu_manager.go:414] "containerMap: removing container" podUID="49719ca9-b7db-4fd3-bcc8-feba390c6874" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993388 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49719ca9-b7db-4fd3-bcc8-feba390c6874" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993395 4468 cpu_manager.go:414] "containerMap: removing container" podUID="6d9cb487-201b-4bc1-ba55-c3e7c4306174" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993400 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d9cb487-201b-4bc1-ba55-c3e7c4306174" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993407 4468 cpu_manager.go:414] "containerMap: removing container" podUID="137214d6-09d8-4a4f-bf37-fcef28d37d07" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993412 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137214d6-09d8-4a4f-bf37-fcef28d37d07" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993420 4468 cpu_manager.go:414] "containerMap: removing container" podUID="ab60f224-4948-4a2b-abf2-116cb1827625" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993425 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab60f224-4948-4a2b-abf2-116cb1827625" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993434 4468 cpu_manager.go:414] "containerMap: removing container" podUID="484b8ef9-46da-48da-a050-97bc7cea4f07" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993438 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="484b8ef9-46da-48da-a050-97bc7cea4f07" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993445 4468 cpu_manager.go:414] "containerMap: removing container" podUID="9002883e-d5ab-4331-b4fa-f3ae000c9784" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993449 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9002883e-d5ab-4331-b4fa-f3ae000c9784" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993458 4468 cpu_manager.go:414] "containerMap: removing container" podUID="22fe1ef7-c64e-4523-be4c-f5460fadf274" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993464 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fe1ef7-c64e-4523-be4c-f5460fadf274" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993476 4468 cpu_manager.go:414] "containerMap: removing container" podUID="42777387-98e9-4718-aa0e-dcdcb64b7a19" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993481 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42777387-98e9-4718-aa0e-dcdcb64b7a19" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993488 4468 cpu_manager.go:414] "containerMap: removing container" podUID="c61a610b-08a7-46ce-b0d5-b2171147362c" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993493 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61a610b-08a7-46ce-b0d5-b2171147362c" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993499 4468 cpu_manager.go:414] "containerMap: removing container" podUID="7c7c9b69-2356-49ba-ad43-2a8773f48a46" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993504 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7c9b69-2356-49ba-ad43-2a8773f48a46" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993511 4468 cpu_manager.go:414] "containerMap: removing container" podUID="4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993515 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993523 4468 cpu_manager.go:414] "containerMap: removing container" podUID="6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993530 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993540 4468 cpu_manager.go:414] "containerMap: removing container" podUID="64cf635f-868a-4021-91aa-f36a314e26c1" containerName="istio-proxy" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993548 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64cf635f-868a-4021-91aa-f36a314e26c1" containerName="istio-proxy" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993558 4468 cpu_manager.go:414] "containerMap: removing container" podUID="5db888ab-7750-41f1-9a4c-3add0f15cc6d" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993566 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db888ab-7750-41f1-9a4c-3add0f15cc6d" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993576 4468 cpu_manager.go:414] "containerMap: removing container" podUID="add4979c-c091-4dea-9853-615756907859" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993583 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add4979c-c091-4dea-9853-615756907859" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993595 4468 cpu_manager.go:414] "containerMap: removing container" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993602 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993612 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3314d4c4-7905-42dd-b342-277da538ba35" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993619 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3314d4c4-7905-42dd-b342-277da538ba35" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993628 4468 cpu_manager.go:414] "containerMap: removing container" podUID="a880cac8-149c-42bc-87c1-31464bf24279" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993632 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a880cac8-149c-42bc-87c1-31464bf24279" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993641 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993646 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993659 4468 cpu_manager.go:414] "containerMap: removing container" podUID="56adb9ab-1a6f-497c-8987-33123b8eb06a" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993667 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56adb9ab-1a6f-497c-8987-33123b8eb06a" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993674 4468 cpu_manager.go:414] "containerMap: removing container" podUID="7f2bdcd0-98ce-4c5b-bb1b-81db35f39186" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993678 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f2bdcd0-98ce-4c5b-bb1b-81db35f39186" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993684 4468 cpu_manager.go:414] "containerMap: removing container" podUID="69e26174-f25d-4904-b2d8-b69a962ca344" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993689 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e26174-f25d-4904-b2d8-b69a962ca344" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993695 4468 cpu_manager.go:414] "containerMap: removing container" podUID="9eb0a51c-3364-487b-8aec-77c2cb215dd7" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993699 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb0a51c-3364-487b-8aec-77c2cb215dd7" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993708 4468 cpu_manager.go:414] "containerMap: removing container" podUID="3314d4c4-7905-42dd-b342-277da538ba35" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993714 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3314d4c4-7905-42dd-b342-277da538ba35" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993726 4468 cpu_manager.go:414] "containerMap: removing container" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993733 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993741 4468 cpu_manager.go:414] "containerMap: removing container" podUID="52833cb7-6657-4225-9bf4-3532bcd3a15f" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993749 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52833cb7-6657-4225-9bf4-3532bcd3a15f" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993758 4468 cpu_manager.go:414] "containerMap: removing container" podUID="ee548a01-d0b9-4387-a699-041ac6e9b205" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993764 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee548a01-d0b9-4387-a699-041ac6e9b205" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993775 4468 cpu_manager.go:414] "containerMap: removing container" podUID="b7a743c1-ed5e-4fec-aa03-e3fb9f10f048" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993782 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7a743c1-ed5e-4fec-aa03-e3fb9f10f048" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993792 4468 cpu_manager.go:414] "containerMap: removing container" podUID="2dbcae46-3576-48d9-9691-50ddfaefd50e" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993800 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dbcae46-3576-48d9-9691-50ddfaefd50e" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993810 4468 cpu_manager.go:414] "containerMap: removing container" podUID="bcddfa21-70dc-407b-803f-c88f7ae0d27e" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993817 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcddfa21-70dc-407b-803f-c88f7ae0d27e" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993827 4468 cpu_manager.go:414] "containerMap: removing container" podUID="ba1e2c95-cd0f-43dc-89be-607ea62fd51d" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993834 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba1e2c95-cd0f-43dc-89be-607ea62fd51d" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993943 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="9002883e-d5ab-4331-b4fa-f3ae000c9784" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993955 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fe1ef7-c64e-4523-be4c-f5460fadf274" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993964 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="42777387-98e9-4718-aa0e-dcdcb64b7a19" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993970 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61a610b-08a7-46ce-b0d5-b2171147362c" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993980 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993987 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="db33d91a-07fb-4c52-832e-0440bc4654af" containerName="istio-proxy" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.993993 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7c9b69-2356-49ba-ad43-2a8773f48a46" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994000 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994005 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994011 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="64cf635f-868a-4021-91aa-f36a314e26c1" containerName="istio-proxy" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994017 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db888ab-7750-41f1-9a4c-3add0f15cc6d" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994023 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="add4979c-c091-4dea-9853-615756907859" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994030 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa877b54-11c9-4574-9d5e-e0d00dff0020" containerName="main" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994036 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb0a51c-3364-487b-8aec-77c2cb215dd7" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994040 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3314d4c4-7905-42dd-b342-277da538ba35" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994045 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a880cac8-149c-42bc-87c1-31464bf24279" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994050 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="137214d6-09d8-4a4f-bf37-fcef28d37d07" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994055 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b2f93b2-9ab8-4f16-afbb-6d5092ac2ba8" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994063 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="56adb9ab-1a6f-497c-8987-33123b8eb06a" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994083 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f2bdcd0-98ce-4c5b-bb1b-81db35f39186" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994094 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e26174-f25d-4904-b2d8-b69a962ca344" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994101 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb0a51c-3364-487b-8aec-77c2cb215dd7" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994107 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d273864b-7bab-4471-8531-3855be5b6182" containerName="main" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994112 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="52833cb7-6657-4225-9bf4-3532bcd3a15f" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994118 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee548a01-d0b9-4387-a699-041ac6e9b205" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994123 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee548a01-d0b9-4387-a699-041ac6e9b205" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994128 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7a743c1-ed5e-4fec-aa03-e3fb9f10f048" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994133 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dbcae46-3576-48d9-9691-50ddfaefd50e" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994138 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcddfa21-70dc-407b-803f-c88f7ae0d27e" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994142 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba1e2c95-cd0f-43dc-89be-607ea62fd51d" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994148 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="af770c09-3be0-418b-8737-3cd556b08035" containerName="maas-api" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994154 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b15b44d-dfe6-4001-966b-f72c56f449c1" containerName="istio-proxy" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994159 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="36dd06a2-cbb4-43df-8411-f3b35e06e98c" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994165 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="de938051-680d-48df-80b7-8e71ee81217b" containerName="cleanup" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994171 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e11991-76bb-4c45-8393-33dbc1ed877f" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994176 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b15b44d-dfe6-4001-966b-f72c56f449c1" containerName="istio-proxy" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994183 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="416eb2ba-b5b4-4e25-a057-d870df27c1f7" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994189 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f68b80b-1983-4fc9-907f-20653341e98a" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994196 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="29813ba6-6304-4a2f-ab9b-ce9f4d904ad5" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994202 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1903b7f6-f2e3-4000-9399-a7117583640f" containerName="main" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994208 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="64964ff4-80ce-4c89-af7b-19d3b24b07f4" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994215 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="065b66a4-c80e-4dde-8bf6-4f14386e6d87" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994221 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9ddb0a-51f8-4d38-9605-74dc25689e46" containerName="payload-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994235 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea95fef4-f1e3-4967-9c5f-b6133ae05eb1" containerName="main" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994239 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bac85e8-7e06-4933-a757-4739e81a0afd" containerName="payload-pre-processing" Oct 30 13:08:11.994497 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994246 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc9a08f6-1257-4e45-9a7d-9687d6554377" containerName="payload-processing" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994253 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="16dae47c-220c-4d83-89e5-52d1746775e8" containerName="istio-proxy" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994258 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b60e79ec-6b0e-4725-81f3-d02c72fc8efb" containerName="istio-proxy" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994264 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf35b28-b756-4281-80d7-bc361b2d56e3" containerName="maas-api" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994269 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778" containerName="cleanup" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994273 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb0a51c-3364-487b-8aec-77c2cb215dd7" containerName="cleanup" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994279 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="e977ddbb-e36d-4056-b775-d91dbbb91881" containerName="maas-api" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994284 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="de938051-680d-48df-80b7-8e71ee81217b" containerName="cleanup" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994290 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="557e9c97-61e1-4358-8249-c79d946c5a9c" containerName="maas-api" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994295 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="de938051-680d-48df-80b7-8e71ee81217b" containerName="cleanup" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994303 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf2843b-7deb-4ffa-8b55-48e59ad1f756" containerName="maas-api" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994311 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e29344f-41fb-4929-aee9-739cf692fa68" containerName="payload-processing" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994317 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="137214d6-09d8-4a4f-bf37-fcef28d37d07" containerName="cleanup" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994323 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b10586a-875c-4812-9bd2-0e65b81d0224" containerName="maas-api" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994329 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae0dc66-24bc-4aad-a275-bc7a8a2c3206" containerName="payload-pre-processing" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994334 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="325f91fe-140a-47b9-8b30-c66476278515" containerName="maas-api" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994339 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1270203a-f11c-4b92-9cb6-b76ab1bbc2a4" containerName="maas-api" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994345 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a87f6f-513a-4a7b-a904-be6b4f218570" containerName="maas-api" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994350 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f76bd7-cafb-48ad-9f99-8a7b88538a85" containerName="payload-pre-processing" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994355 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3314d4c4-7905-42dd-b342-277da538ba35" containerName="cleanup" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994360 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="5929f380-6cc0-4ccf-97b8-a3e54b01430f" containerName="discovery" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994365 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca7e7d8-d36f-4378-8052-04a62530fff6" containerName="maas-api" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994370 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="110bd521-953f-48c7-990a-823b164a281b" containerName="payload-pre-processing" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994377 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd5da4da-cc71-4a46-a697-396a96373776" containerName="payload-processing" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994387 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="49719ca9-b7db-4fd3-bcc8-feba390c6874" containerName="payload-processing" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994394 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d9cb487-201b-4bc1-ba55-c3e7c4306174" containerName="payload-pre-processing" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994400 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="137214d6-09d8-4a4f-bf37-fcef28d37d07" containerName="cleanup" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994407 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab60f224-4948-4a2b-abf2-116cb1827625" containerName="payload-processing" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.994413 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="484b8ef9-46da-48da-a050-97bc7cea4f07" containerName="payload-processing" Oct 30 13:08:12.000866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:11.997734 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf" Oct 30 13:08:12.016323 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.016298 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf"] Oct 30 13:08:12.017817 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.017793 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:08:12.017922 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.017858 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:08:12.025300 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.025274 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-maas-api-tls\") pod \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\" (UID: \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\") " Oct 30 13:08:12.025402 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.025362 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-kube-api-access-d8pxg\" (UniqueName: \"kubernetes.io/projected/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-kube-api-access-d8pxg\") pod \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\" (UID: \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\") " Oct 30 13:08:12.025482 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.025449 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-metrics-certs\") pod \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\" (UID: \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\") " Oct 30 13:08:12.025547 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.025514 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-openshift-service-ca\") pod \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\" (UID: \"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd\") " Oct 30 13:08:12.026046 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.026012 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-openshift-service-ca" pod "4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd" (UID: "4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:12.027587 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.027560 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-maas-api-tls" pod "4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd" (UID: "4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:12.027694 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.027682 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-metrics-certs" pod "4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd" (UID: "4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:12.027790 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.027773 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-kube-api-access-d8pxg" pod "4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd" (UID: "4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd"). InnerVolumeSpecName "kube-api-access-d8pxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:12.126525 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.126490 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2qwt\" (UniqueName: \"kubernetes.io/projected/2bf6ae69-b68b-402b-bbd4-0846700c8962-kube-api-access-f2qwt\") pod \"maas-api-revoke-sub-4e9110a7-tslmf\" (UID: \"2bf6ae69-b68b-402b-bbd4-0846700c8962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf" Oct 30 13:08:12.126677 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.126565 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf6ae69-b68b-402b-bbd4-0846700c8962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e9110a7-tslmf\" (UID: \"2bf6ae69-b68b-402b-bbd4-0846700c8962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf" Oct 30 13:08:12.126748 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.126692 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf6ae69-b68b-402b-bbd4-0846700c8962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e9110a7-tslmf\" (UID: \"2bf6ae69-b68b-402b-bbd4-0846700c8962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf" Oct 30 13:08:12.126803 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.126781 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:12.126861 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.126799 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:12.126861 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.126814 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8pxg\" (UniqueName: \"kubernetes.io/projected/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-kube-api-access-d8pxg\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:12.126861 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.126829 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:12.227519 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.227489 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2qwt\" (UniqueName: \"kubernetes.io/projected/2bf6ae69-b68b-402b-bbd4-0846700c8962-kube-api-access-f2qwt\") pod \"maas-api-revoke-sub-4e9110a7-tslmf\" (UID: \"2bf6ae69-b68b-402b-bbd4-0846700c8962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf" Oct 30 13:08:12.227740 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.227709 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf6ae69-b68b-402b-bbd4-0846700c8962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e9110a7-tslmf\" (UID: \"2bf6ae69-b68b-402b-bbd4-0846700c8962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf" Oct 30 13:08:12.227918 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.227902 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf6ae69-b68b-402b-bbd4-0846700c8962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e9110a7-tslmf\" (UID: \"2bf6ae69-b68b-402b-bbd4-0846700c8962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf" Oct 30 13:08:12.228383 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.228362 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf6ae69-b68b-402b-bbd4-0846700c8962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e9110a7-tslmf\" (UID: \"2bf6ae69-b68b-402b-bbd4-0846700c8962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf" Oct 30 13:08:12.230168 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.230147 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf6ae69-b68b-402b-bbd4-0846700c8962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e9110a7-tslmf\" (UID: \"2bf6ae69-b68b-402b-bbd4-0846700c8962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf" Oct 30 13:08:12.236387 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.236367 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" Oct 30 13:08:12.254684 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.254661 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2qwt\" (UniqueName: \"kubernetes.io/projected/2bf6ae69-b68b-402b-bbd4-0846700c8962-kube-api-access-f2qwt\") pod \"maas-api-revoke-sub-4e9110a7-tslmf\" (UID: \"2bf6ae69-b68b-402b-bbd4-0846700c8962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf" Oct 30 13:08:12.329391 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.329298 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35ef76d8-4829-4123-bbff-e7b36c59afba-kube-api-access-rvdml\" (UniqueName: \"kubernetes.io/projected/35ef76d8-4829-4123-bbff-e7b36c59afba-kube-api-access-rvdml\") pod \"35ef76d8-4829-4123-bbff-e7b36c59afba\" (UID: \"35ef76d8-4829-4123-bbff-e7b36c59afba\") " Oct 30 13:08:12.329532 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.329461 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35ef76d8-4829-4123-bbff-e7b36c59afba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ef76d8-4829-4123-bbff-e7b36c59afba-plugins-config-volume\") pod \"35ef76d8-4829-4123-bbff-e7b36c59afba\" (UID: \"35ef76d8-4829-4123-bbff-e7b36c59afba\") " Oct 30 13:08:12.329897 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.329866 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35ef76d8-4829-4123-bbff-e7b36c59afba-plugins-config-volume" pod "35ef76d8-4829-4123-bbff-e7b36c59afba" (UID: "35ef76d8-4829-4123-bbff-e7b36c59afba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:12.331582 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.331554 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ef76d8-4829-4123-bbff-e7b36c59afba-kube-api-access-rvdml" pod "35ef76d8-4829-4123-bbff-e7b36c59afba" (UID: "35ef76d8-4829-4123-bbff-e7b36c59afba"). InnerVolumeSpecName "kube-api-access-rvdml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:12.377441 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.377413 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf" Oct 30 13:08:12.434388 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.433259 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ef76d8-4829-4123-bbff-e7b36c59afba-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:12.434388 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.433290 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvdml\" (UniqueName: \"kubernetes.io/projected/35ef76d8-4829-4123-bbff-e7b36c59afba-kube-api-access-rvdml\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:12.545281 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.545245 4468 generic.go:356] "Generic (PLEG): container finished" podID="35ef76d8-4829-4123-bbff-e7b36c59afba" containerID="6b96d53841725417ee5ab9a88eaf75c34a158c87944593ff48235f3d51639829" exitCode=0 Oct 30 13:08:12.545907 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.545297 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" Oct 30 13:08:12.545907 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.545401 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" event={"ID":"35ef76d8-4829-4123-bbff-e7b36c59afba","Type":"ContainerDied","Data":"6b96d53841725417ee5ab9a88eaf75c34a158c87944593ff48235f3d51639829"} Oct 30 13:08:12.545907 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.545452 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm" event={"ID":"35ef76d8-4829-4123-bbff-e7b36c59afba","Type":"ContainerDied","Data":"0f45f32429dba607812dc922f7499da81f055ee505786c1ee866b7c391d5c613"} Oct 30 13:08:12.545907 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.545485 4468 scope.go:122] "RemoveContainer" containerID="6b96d53841725417ee5ab9a88eaf75c34a158c87944593ff48235f3d51639829" Oct 30 13:08:12.546898 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.546880 4468 generic.go:356] "Generic (PLEG): container finished" podID="4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd" containerID="06ab4a4bd6c138d58f54bba8f63c4d20a8942f2e25f2b94bb94c24bb79eb606e" exitCode=0 Oct 30 13:08:12.546971 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.546909 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" event={"ID":"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd","Type":"ContainerDied","Data":"06ab4a4bd6c138d58f54bba8f63c4d20a8942f2e25f2b94bb94c24bb79eb606e"} Oct 30 13:08:12.546971 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.546936 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" event={"ID":"4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd","Type":"ContainerDied","Data":"0a1fe9ec60f1223cd7e9f1ef6d9a0ee2eb1b237a262f12e79fbc418b4ecf13fd"} Oct 30 13:08:12.547132 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.546986 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p" Oct 30 13:08:12.555948 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.555926 4468 scope.go:122] "RemoveContainer" containerID="6b96d53841725417ee5ab9a88eaf75c34a158c87944593ff48235f3d51639829" Oct 30 13:08:12.556243 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:08:12.556222 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b96d53841725417ee5ab9a88eaf75c34a158c87944593ff48235f3d51639829\": container with ID starting with 6b96d53841725417ee5ab9a88eaf75c34a158c87944593ff48235f3d51639829 not found: ID does not exist" containerID="6b96d53841725417ee5ab9a88eaf75c34a158c87944593ff48235f3d51639829" Oct 30 13:08:12.556315 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.556250 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b96d53841725417ee5ab9a88eaf75c34a158c87944593ff48235f3d51639829"} err="failed to get container status \"6b96d53841725417ee5ab9a88eaf75c34a158c87944593ff48235f3d51639829\": rpc error: code = NotFound desc = could not find container \"6b96d53841725417ee5ab9a88eaf75c34a158c87944593ff48235f3d51639829\": container with ID starting with 6b96d53841725417ee5ab9a88eaf75c34a158c87944593ff48235f3d51639829 not found: ID does not exist" Oct 30 13:08:12.556315 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.556265 4468 scope.go:122] "RemoveContainer" containerID="06ab4a4bd6c138d58f54bba8f63c4d20a8942f2e25f2b94bb94c24bb79eb606e" Oct 30 13:08:12.572741 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.572722 4468 scope.go:122] "RemoveContainer" containerID="06ab4a4bd6c138d58f54bba8f63c4d20a8942f2e25f2b94bb94c24bb79eb606e" Oct 30 13:08:12.573049 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:08:12.573028 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ab4a4bd6c138d58f54bba8f63c4d20a8942f2e25f2b94bb94c24bb79eb606e\": container with ID starting with 06ab4a4bd6c138d58f54bba8f63c4d20a8942f2e25f2b94bb94c24bb79eb606e not found: ID does not exist" containerID="06ab4a4bd6c138d58f54bba8f63c4d20a8942f2e25f2b94bb94c24bb79eb606e" Oct 30 13:08:12.573122 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.573055 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ab4a4bd6c138d58f54bba8f63c4d20a8942f2e25f2b94bb94c24bb79eb606e"} err="failed to get container status \"06ab4a4bd6c138d58f54bba8f63c4d20a8942f2e25f2b94bb94c24bb79eb606e\": rpc error: code = NotFound desc = could not find container \"06ab4a4bd6c138d58f54bba8f63c4d20a8942f2e25f2b94bb94c24bb79eb606e\": container with ID starting with 06ab4a4bd6c138d58f54bba8f63c4d20a8942f2e25f2b94bb94c24bb79eb606e not found: ID does not exist" Oct 30 13:08:12.603221 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.603199 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm"] Oct 30 13:08:12.609958 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.609935 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9d45fb-bf6d6c6bc-ngpbm"] Oct 30 13:08:12.645795 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.645772 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p"] Oct 30 13:08:12.655009 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.654988 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-327e31-79f4bb5656-bm85p"] Oct 30 13:08:12.687210 ip-10-0-131-76 kubenswrapper[4468]: W1030 13:08:12.687186 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf6ae69_b68b_402b_bbd4_0846700c8962.slice/crio-05dbb2a67e8fcce9056116d83397afd19c666c108c371c1d1e2d1137a37be533 WatchSource:0}: Error finding container 05dbb2a67e8fcce9056116d83397afd19c666c108c371c1d1e2d1137a37be533: Status 404 returned error can't find the container with id 05dbb2a67e8fcce9056116d83397afd19c666c108c371c1d1e2d1137a37be533 Oct 30 13:08:12.689945 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:12.689924 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf"] Oct 30 13:08:13.088286 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:13.088255 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35ef76d8-4829-4123-bbff-e7b36c59afba" path="/var/lib/kubelet/pods/35ef76d8-4829-4123-bbff-e7b36c59afba/volumes" Oct 30 13:08:13.088741 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:13.088723 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd" path="/var/lib/kubelet/pods/4f9fd4ee-b6d4-4c72-9a09-3e0bf849e3cd/volumes" Oct 30 13:08:13.089190 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:13.089175 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e11991-76bb-4c45-8393-33dbc1ed877f" path="/var/lib/kubelet/pods/e8e11991-76bb-4c45-8393-33dbc1ed877f/volumes" Oct 30 13:08:13.553348 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:13.553309 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf" event={"ID":"2bf6ae69-b68b-402b-bbd4-0846700c8962","Type":"ContainerStarted","Data":"d1c63eeb23207f1a73c1c52a22fcb8a06d2c073343c871f4c87362bd3441f5a3"} Oct 30 13:08:13.553793 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:13.553372 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf" event={"ID":"2bf6ae69-b68b-402b-bbd4-0846700c8962","Type":"ContainerStarted","Data":"05dbb2a67e8fcce9056116d83397afd19c666c108c371c1d1e2d1137a37be533"} Oct 30 13:08:13.605592 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:13.605548 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf" podStartSLOduration=2.60553546 podStartE2EDuration="2.60553546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:13.602524006 +0000 UTC m=+4444.636662403" watchObservedRunningTime="2026-10-30 13:08:13.60553546 +0000 UTC m=+4444.639673887" Oct 30 13:08:14.532942 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.532919 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 13:08:14.562381 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.562323 4468 generic.go:356] "Generic (PLEG): container finished" podID="b77483d1-fb3c-4be6-8d49-c6863bfcc84d" containerID="4034c93f561b1ef45372c69924d5859fc11ad2be3b45f227d1eed9c1f21e5184" exitCode=0 Oct 30 13:08:14.562381 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.562366 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" Oct 30 13:08:14.562792 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.562397 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" event={"ID":"b77483d1-fb3c-4be6-8d49-c6863bfcc84d","Type":"ContainerDied","Data":"4034c93f561b1ef45372c69924d5859fc11ad2be3b45f227d1eed9c1f21e5184"} Oct 30 13:08:14.562792 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.562449 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r" event={"ID":"b77483d1-fb3c-4be6-8d49-c6863bfcc84d","Type":"ContainerDied","Data":"0ce08c13dd33b4a0f23e9421ec666bbaf2fc58760a6e2f4d4980cd9d496abdf3"} Oct 30 13:08:14.562792 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.562479 4468 scope.go:122] "RemoveContainer" containerID="4034c93f561b1ef45372c69924d5859fc11ad2be3b45f227d1eed9c1f21e5184" Oct 30 13:08:14.563943 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.563923 4468 generic.go:356] "Generic (PLEG): container finished" podID="2bf6ae69-b68b-402b-bbd4-0846700c8962" containerID="d1c63eeb23207f1a73c1c52a22fcb8a06d2c073343c871f4c87362bd3441f5a3" exitCode=0 Oct 30 13:08:14.564045 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.564012 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf" event={"ID":"2bf6ae69-b68b-402b-bbd4-0846700c8962","Type":"ContainerDied","Data":"d1c63eeb23207f1a73c1c52a22fcb8a06d2c073343c871f4c87362bd3441f5a3"} Oct 30 13:08:14.571409 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.571392 4468 scope.go:122] "RemoveContainer" containerID="4034c93f561b1ef45372c69924d5859fc11ad2be3b45f227d1eed9c1f21e5184" Oct 30 13:08:14.571651 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:08:14.571629 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4034c93f561b1ef45372c69924d5859fc11ad2be3b45f227d1eed9c1f21e5184\": container with ID starting with 4034c93f561b1ef45372c69924d5859fc11ad2be3b45f227d1eed9c1f21e5184 not found: ID does not exist" containerID="4034c93f561b1ef45372c69924d5859fc11ad2be3b45f227d1eed9c1f21e5184" Oct 30 13:08:14.571705 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.571658 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4034c93f561b1ef45372c69924d5859fc11ad2be3b45f227d1eed9c1f21e5184"} err="failed to get container status \"4034c93f561b1ef45372c69924d5859fc11ad2be3b45f227d1eed9c1f21e5184\": rpc error: code = NotFound desc = could not find container \"4034c93f561b1ef45372c69924d5859fc11ad2be3b45f227d1eed9c1f21e5184\": container with ID starting with 4034c93f561b1ef45372c69924d5859fc11ad2be3b45f227d1eed9c1f21e5184 not found: ID does not exist" Oct 30 13:08:14.655000 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.654974 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istiod-ca-cert\") pod \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " Oct 30 13:08:14.655138 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.655029 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-credential-socket\") pod \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " Oct 30 13:08:14.655222 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.655205 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-kube-api-access-dcmzj\" (UniqueName: \"kubernetes.io/projected/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-kube-api-access-dcmzj\") pod \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " Oct 30 13:08:14.655298 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.655247 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-workload-socket\") pod \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " Oct 30 13:08:14.655298 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.655290 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-data\") pod \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " Oct 30 13:08:14.655408 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.655331 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-token\" (UniqueName: \"kubernetes.io/projected/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-token\") pod \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " Oct 30 13:08:14.655408 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.655335 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-credential-socket" pod "b77483d1-fb3c-4be6-8d49-c6863bfcc84d" (UID: "b77483d1-fb3c-4be6-8d49-c6863bfcc84d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:14.655408 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.655390 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-envoy\") pod \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " Oct 30 13:08:14.655569 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.655425 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istiod-ca-cert" pod "b77483d1-fb3c-4be6-8d49-c6863bfcc84d" (UID: "b77483d1-fb3c-4be6-8d49-c6863bfcc84d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:14.655569 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.655444 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-workload-certs\") pod \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " Oct 30 13:08:14.655569 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.655492 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-podinfo\") pod \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\" (UID: \"b77483d1-fb3c-4be6-8d49-c6863bfcc84d\") " Oct 30 13:08:14.655716 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.655571 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-workload-socket" pod "b77483d1-fb3c-4be6-8d49-c6863bfcc84d" (UID: "b77483d1-fb3c-4be6-8d49-c6863bfcc84d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:14.655716 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.655585 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-data" pod "b77483d1-fb3c-4be6-8d49-c6863bfcc84d" (UID: "b77483d1-fb3c-4be6-8d49-c6863bfcc84d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:14.655812 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.655751 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-workload-certs" pod "b77483d1-fb3c-4be6-8d49-c6863bfcc84d" (UID: "b77483d1-fb3c-4be6-8d49-c6863bfcc84d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:14.655975 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.655954 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:14.656052 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.655981 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:14.656052 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.655995 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:14.656052 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.656010 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:14.656052 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.656024 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:14.657556 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.657532 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-token" pod "b77483d1-fb3c-4be6-8d49-c6863bfcc84d" (UID: "b77483d1-fb3c-4be6-8d49-c6863bfcc84d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:14.657734 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.657563 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-kube-api-access-dcmzj" pod "b77483d1-fb3c-4be6-8d49-c6863bfcc84d" (UID: "b77483d1-fb3c-4be6-8d49-c6863bfcc84d"). InnerVolumeSpecName "kube-api-access-dcmzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:14.657822 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.657806 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-podinfo" pod "b77483d1-fb3c-4be6-8d49-c6863bfcc84d" (UID: "b77483d1-fb3c-4be6-8d49-c6863bfcc84d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:08:14.657878 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.657824 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-envoy" pod "b77483d1-fb3c-4be6-8d49-c6863bfcc84d" (UID: "b77483d1-fb3c-4be6-8d49-c6863bfcc84d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:14.757473 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.757442 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcmzj\" (UniqueName: \"kubernetes.io/projected/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-kube-api-access-dcmzj\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:14.757473 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.757468 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:14.757473 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.757478 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:14.757686 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.757487 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b77483d1-fb3c-4be6-8d49-c6863bfcc84d-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:14.913174 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.913139 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r"] Oct 30 13:08:14.926550 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:14.926525 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-c2h2r"] Oct 30 13:08:15.090247 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:15.090217 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b77483d1-fb3c-4be6-8d49-c6863bfcc84d" path="/var/lib/kubelet/pods/b77483d1-fb3c-4be6-8d49-c6863bfcc84d/volumes" Oct 30 13:08:15.700625 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:15.700595 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf" Oct 30 13:08:15.866773 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:15.866708 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf6ae69-b68b-402b-bbd4-0846700c8962-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf6ae69-b68b-402b-bbd4-0846700c8962-maas-api-openshift-ca\") pod \"2bf6ae69-b68b-402b-bbd4-0846700c8962\" (UID: \"2bf6ae69-b68b-402b-bbd4-0846700c8962\") " Oct 30 13:08:15.866773 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:15.866757 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf6ae69-b68b-402b-bbd4-0846700c8962-kube-api-access-f2qwt\" (UniqueName: \"kubernetes.io/projected/2bf6ae69-b68b-402b-bbd4-0846700c8962-kube-api-access-f2qwt\") pod \"2bf6ae69-b68b-402b-bbd4-0846700c8962\" (UID: \"2bf6ae69-b68b-402b-bbd4-0846700c8962\") " Oct 30 13:08:15.866983 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:15.866789 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bf6ae69-b68b-402b-bbd4-0846700c8962-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf6ae69-b68b-402b-bbd4-0846700c8962-maas-api-xks-ca\") pod \"2bf6ae69-b68b-402b-bbd4-0846700c8962\" (UID: \"2bf6ae69-b68b-402b-bbd4-0846700c8962\") " Oct 30 13:08:15.867158 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:15.867128 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf6ae69-b68b-402b-bbd4-0846700c8962-maas-api-openshift-ca" pod "2bf6ae69-b68b-402b-bbd4-0846700c8962" (UID: "2bf6ae69-b68b-402b-bbd4-0846700c8962"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:15.867288 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:15.867241 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf6ae69-b68b-402b-bbd4-0846700c8962-maas-api-openshift-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:15.868847 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:15.868824 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf6ae69-b68b-402b-bbd4-0846700c8962-maas-api-xks-ca" pod "2bf6ae69-b68b-402b-bbd4-0846700c8962" (UID: "2bf6ae69-b68b-402b-bbd4-0846700c8962"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:15.868936 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:15.868833 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf6ae69-b68b-402b-bbd4-0846700c8962-kube-api-access-f2qwt" pod "2bf6ae69-b68b-402b-bbd4-0846700c8962" (UID: "2bf6ae69-b68b-402b-bbd4-0846700c8962"). InnerVolumeSpecName "kube-api-access-f2qwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:15.968738 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:15.968715 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2qwt\" (UniqueName: \"kubernetes.io/projected/2bf6ae69-b68b-402b-bbd4-0846700c8962-kube-api-access-f2qwt\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:15.968738 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:15.968742 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf6ae69-b68b-402b-bbd4-0846700c8962-maas-api-xks-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:16.576746 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:16.576718 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf" Oct 30 13:08:16.576933 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:16.576749 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e9110a7-tslmf" event={"ID":"2bf6ae69-b68b-402b-bbd4-0846700c8962","Type":"ContainerDied","Data":"05dbb2a67e8fcce9056116d83397afd19c666c108c371c1d1e2d1137a37be533"} Oct 30 13:08:16.576933 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:16.576820 4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05dbb2a67e8fcce9056116d83397afd19c666c108c371c1d1e2d1137a37be533" Oct 30 13:08:16.813806 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:16.813771 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" podUID="4572318d-2ff5-43d0-9071-faeb6e94c178" containerName="payload-processing" containerID="cri-o://707155a85afee81f8881e08418e961fccbb2123b7f25ff1b8d0e34d273172f56" gracePeriod=25 Oct 30 13:08:16.845633 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:16.845574 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" podUID="127e9882-c84b-4162-9565-834a8126f9e3" containerName="payload-pre-processing" containerID="cri-o://645e122f975e1e42c9c9766f80a4b290047cb9b7e75429a172953a51a91663a8" gracePeriod=25 Oct 30 13:08:20.585585 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:20.585548 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7"] Oct 30 13:08:20.771640 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:20.771591 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" containerID="cri-o://535f0611367523074cbbff213269aa991b138aa3e3b8078e9e10375037f26231" gracePeriod=45 Oct 30 13:08:20.893916 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:20.893886 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s"] Oct 30 13:08:20.894182 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:20.894156 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" podUID="4027e387-9043-4303-a6e1-c18365dcfac0" containerName="istio-proxy" containerID="cri-o://dd308d1b1bbc57bbb94ec515a3ecdb0ef44846bd3f779e1b52da259b1970a841" gracePeriod=30 Oct 30 13:08:21.024596 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.024568 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 13:08:21.220780 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.220746 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-tmp-dir\") pod \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " Oct 30 13:08:21.221004 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.220804 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-tls-certs\" (UniqueName: \"kubernetes.io/secret/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-tls-certs\") pod \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " Oct 30 13:08:21.221004 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.220854 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-model-cache\") pod \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " Oct 30 13:08:21.221161 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.221027 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-tmp-dir" pod "76e7b4b9-e1bf-4cd7-9137-1108398cc50e" (UID: "76e7b4b9-e1bf-4cd7-9137-1108398cc50e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:21.221223 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.221170 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-dshm\") pod \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " Oct 30 13:08:21.221283 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.221170 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-model-cache" pod "76e7b4b9-e1bf-4cd7-9137-1108398cc50e" (UID: "76e7b4b9-e1bf-4cd7-9137-1108398cc50e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:21.221339 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.221262 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-kube-api-access-9bs9n\" (UniqueName: \"kubernetes.io/projected/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-kube-api-access-9bs9n\") pod \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " Oct 30 13:08:21.221441 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.221425 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-home\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-home\") pod \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\" (UID: \"76e7b4b9-e1bf-4cd7-9137-1108398cc50e\") " Oct 30 13:08:21.221929 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.221742 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-home" pod "76e7b4b9-e1bf-4cd7-9137-1108398cc50e" (UID: "76e7b4b9-e1bf-4cd7-9137-1108398cc50e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:21.221929 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.221837 4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-home\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.221929 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.221877 4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-tmp-dir\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.221929 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.221892 4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-model-cache\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.223742 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.223700 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-tls-certs" pod "76e7b4b9-e1bf-4cd7-9137-1108398cc50e" (UID: "76e7b4b9-e1bf-4cd7-9137-1108398cc50e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:21.223846 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.223791 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-dshm" pod "76e7b4b9-e1bf-4cd7-9137-1108398cc50e" (UID: "76e7b4b9-e1bf-4cd7-9137-1108398cc50e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:21.223846 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.223825 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-kube-api-access-9bs9n" pod "76e7b4b9-e1bf-4cd7-9137-1108398cc50e" (UID: "76e7b4b9-e1bf-4cd7-9137-1108398cc50e"). InnerVolumeSpecName "kube-api-access-9bs9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:21.322641 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.322596 4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-dshm\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.322641 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.322637 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bs9n\" (UniqueName: \"kubernetes.io/projected/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-kube-api-access-9bs9n\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.322893 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.322650 4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e7b4b9-e1bf-4cd7-9137-1108398cc50e-tls-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.609615 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.609528 4468 generic.go:356] "Generic (PLEG): container finished" podID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerID="535f0611367523074cbbff213269aa991b138aa3e3b8078e9e10375037f26231" exitCode=0 Oct 30 13:08:21.609615 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.609559 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" Oct 30 13:08:21.609615 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.609593 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" event={"ID":"76e7b4b9-e1bf-4cd7-9137-1108398cc50e","Type":"ContainerDied","Data":"535f0611367523074cbbff213269aa991b138aa3e3b8078e9e10375037f26231"} Oct 30 13:08:21.610218 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.609643 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" event={"ID":"76e7b4b9-e1bf-4cd7-9137-1108398cc50e","Type":"ContainerDied","Data":"0d56d5b77ff8aba2535618e6cbc6c2b28b724616fd03f70eec333acae7a4b79c"} Oct 30 13:08:21.610218 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.609665 4468 scope.go:122] "RemoveContainer" containerID="535f0611367523074cbbff213269aa991b138aa3e3b8078e9e10375037f26231" Oct 30 13:08:21.624119 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.624095 4468 scope.go:122] "RemoveContainer" containerID="535f0611367523074cbbff213269aa991b138aa3e3b8078e9e10375037f26231" Oct 30 13:08:21.624375 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:08:21.624355 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535f0611367523074cbbff213269aa991b138aa3e3b8078e9e10375037f26231\": container with ID starting with 535f0611367523074cbbff213269aa991b138aa3e3b8078e9e10375037f26231 not found: ID does not exist" containerID="535f0611367523074cbbff213269aa991b138aa3e3b8078e9e10375037f26231" Oct 30 13:08:21.624445 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.624383 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535f0611367523074cbbff213269aa991b138aa3e3b8078e9e10375037f26231"} err="failed to get container status \"535f0611367523074cbbff213269aa991b138aa3e3b8078e9e10375037f26231\": rpc error: code = NotFound desc = could not find container \"535f0611367523074cbbff213269aa991b138aa3e3b8078e9e10375037f26231\": container with ID starting with 535f0611367523074cbbff213269aa991b138aa3e3b8078e9e10375037f26231 not found: ID does not exist" Oct 30 13:08:21.677852 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.677825 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq"] Oct 30 13:08:21.681823 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.681799 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq"] Oct 30 13:08:21.971163 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:21.971129 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/distinct-w2-47fd99-kserve-66b6c4d966-8t5tq" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:23.088783 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.088751 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" path="/var/lib/kubelet/pods/76e7b4b9-e1bf-4cd7-9137-1108398cc50e/volumes" Oct 30 13:08:23.125103 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125059 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c99c3aa-rt7jk"] Oct 30 13:08:23.125493 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125481 4468 cpu_manager.go:414] "containerMap: removing container" podUID="6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778" containerName="cleanup" Oct 30 13:08:23.125545 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125495 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf3ab9f-f6f2-4d98-9f6a-1af9c6b7f778" containerName="cleanup" Oct 30 13:08:23.125545 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125519 4468 cpu_manager.go:414] "containerMap: removing container" podUID="b77483d1-fb3c-4be6-8d49-c6863bfcc84d" containerName="istio-proxy" Oct 30 13:08:23.125545 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125525 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b77483d1-fb3c-4be6-8d49-c6863bfcc84d" containerName="istio-proxy" Oct 30 13:08:23.125545 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125534 4468 cpu_manager.go:414] "containerMap: removing container" podUID="ee548a01-d0b9-4387-a699-041ac6e9b205" containerName="cleanup" Oct 30 13:08:23.125545 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125540 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee548a01-d0b9-4387-a699-041ac6e9b205" containerName="cleanup" Oct 30 13:08:23.125545 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125547 4468 cpu_manager.go:414] "containerMap: removing container" podUID="35ef76d8-4829-4123-bbff-e7b36c59afba" containerName="payload-pre-processing" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125552 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35ef76d8-4829-4123-bbff-e7b36c59afba" containerName="payload-pre-processing" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125563 4468 cpu_manager.go:414] "containerMap: removing container" podUID="db33d91a-07fb-4c52-832e-0440bc4654af" containerName="istio-proxy" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125568 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db33d91a-07fb-4c52-832e-0440bc4654af" containerName="istio-proxy" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125576 4468 cpu_manager.go:414] "containerMap: removing container" podUID="9eb0a51c-3364-487b-8aec-77c2cb215dd7" containerName="cleanup" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125581 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb0a51c-3364-487b-8aec-77c2cb215dd7" containerName="cleanup" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125587 4468 cpu_manager.go:414] "containerMap: removing container" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125592 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125599 4468 cpu_manager.go:414] "containerMap: removing container" podUID="de938051-680d-48df-80b7-8e71ee81217b" containerName="cleanup" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125603 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de938051-680d-48df-80b7-8e71ee81217b" containerName="cleanup" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125611 4468 cpu_manager.go:414] "containerMap: removing container" podUID="137214d6-09d8-4a4f-bf37-fcef28d37d07" containerName="cleanup" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125615 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137214d6-09d8-4a4f-bf37-fcef28d37d07" containerName="cleanup" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125622 4468 cpu_manager.go:414] "containerMap: removing container" podUID="2bf6ae69-b68b-402b-bbd4-0846700c8962" containerName="revoke-keys" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125627 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf6ae69-b68b-402b-bbd4-0846700c8962" containerName="revoke-keys" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125684 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e7b4b9-e1bf-4cd7-9137-1108398cc50e" containerName="main" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125694 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf6ae69-b68b-402b-bbd4-0846700c8962" containerName="revoke-keys" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125701 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="db33d91a-07fb-4c52-832e-0440bc4654af" containerName="istio-proxy" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125707 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3314d4c4-7905-42dd-b342-277da538ba35" containerName="cleanup" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125714 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee548a01-d0b9-4387-a699-041ac6e9b205" containerName="cleanup" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125719 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b77483d1-fb3c-4be6-8d49-c6863bfcc84d" containerName="istio-proxy" Oct 30 13:08:23.125743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.125725 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="35ef76d8-4829-4123-bbff-e7b36c59afba" containerName="payload-pre-processing" Oct 30 13:08:23.130445 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.130424 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c99c3aa-rt7jk" Oct 30 13:08:23.140169 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.140139 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80712f3d-cc77-4955-86ce-3c66d7aac999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c99c3aa-rt7jk\" (UID: \"80712f3d-cc77-4955-86ce-3c66d7aac999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c99c3aa-rt7jk" Oct 30 13:08:23.140286 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.140186 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80712f3d-cc77-4955-86ce-3c66d7aac999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c99c3aa-rt7jk\" (UID: \"80712f3d-cc77-4955-86ce-3c66d7aac999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c99c3aa-rt7jk" Oct 30 13:08:23.140338 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.140320 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2tn6\" (UniqueName: \"kubernetes.io/projected/80712f3d-cc77-4955-86ce-3c66d7aac999-kube-api-access-q2tn6\") pod \"maas-api-revoke-sub-6c99c3aa-rt7jk\" (UID: \"80712f3d-cc77-4955-86ce-3c66d7aac999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c99c3aa-rt7jk" Oct 30 13:08:23.152411 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.152384 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c99c3aa-rt7jk"] Oct 30 13:08:23.162118 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.162061 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:08:23.162216 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.162138 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:08:23.241729 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.241702 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80712f3d-cc77-4955-86ce-3c66d7aac999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c99c3aa-rt7jk\" (UID: \"80712f3d-cc77-4955-86ce-3c66d7aac999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c99c3aa-rt7jk" Oct 30 13:08:23.241890 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.241745 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80712f3d-cc77-4955-86ce-3c66d7aac999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c99c3aa-rt7jk\" (UID: \"80712f3d-cc77-4955-86ce-3c66d7aac999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c99c3aa-rt7jk" Oct 30 13:08:23.241890 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.241779 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2tn6\" (UniqueName: \"kubernetes.io/projected/80712f3d-cc77-4955-86ce-3c66d7aac999-kube-api-access-q2tn6\") pod \"maas-api-revoke-sub-6c99c3aa-rt7jk\" (UID: \"80712f3d-cc77-4955-86ce-3c66d7aac999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c99c3aa-rt7jk" Oct 30 13:08:23.242519 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.242491 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80712f3d-cc77-4955-86ce-3c66d7aac999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c99c3aa-rt7jk\" (UID: \"80712f3d-cc77-4955-86ce-3c66d7aac999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c99c3aa-rt7jk" Oct 30 13:08:23.244029 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.244007 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80712f3d-cc77-4955-86ce-3c66d7aac999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c99c3aa-rt7jk\" (UID: \"80712f3d-cc77-4955-86ce-3c66d7aac999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c99c3aa-rt7jk" Oct 30 13:08:23.266739 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.266710 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2tn6\" (UniqueName: \"kubernetes.io/projected/80712f3d-cc77-4955-86ce-3c66d7aac999-kube-api-access-q2tn6\") pod \"maas-api-revoke-sub-6c99c3aa-rt7jk\" (UID: \"80712f3d-cc77-4955-86ce-3c66d7aac999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c99c3aa-rt7jk" Oct 30 13:08:23.443002 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.442977 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c99c3aa-rt7jk" Oct 30 13:08:23.603619 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:08:23.603573 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:08:23.746476 ip-10-0-131-76 kubenswrapper[4468]: W1030 13:08:23.746442 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80712f3d_cc77_4955_86ce_3c66d7aac999.slice/crio-35d99edd7a84d7f65eeb790750ef96faed724c1a347239a584ab6e8612b042f3 WatchSource:0}: Error finding container 35d99edd7a84d7f65eeb790750ef96faed724c1a347239a584ab6e8612b042f3: Status 404 returned error can't find the container with id 35d99edd7a84d7f65eeb790750ef96faed724c1a347239a584ab6e8612b042f3 Oct 30 13:08:23.747644 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:23.747620 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c99c3aa-rt7jk"] Oct 30 13:08:24.631414 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:24.631384 4468 generic.go:356] "Generic (PLEG): container finished" podID="80712f3d-cc77-4955-86ce-3c66d7aac999" containerID="6ff40cbb51ecd4726fc47662cd76c32778703a6606c4b81f5fa499a70d752e9b" exitCode=0 Oct 30 13:08:24.631733 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:24.631438 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c99c3aa-rt7jk" event={"ID":"80712f3d-cc77-4955-86ce-3c66d7aac999","Type":"ContainerDied","Data":"6ff40cbb51ecd4726fc47662cd76c32778703a6606c4b81f5fa499a70d752e9b"} Oct 30 13:08:24.631733 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:24.631484 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c99c3aa-rt7jk" event={"ID":"80712f3d-cc77-4955-86ce-3c66d7aac999","Type":"ContainerStarted","Data":"35d99edd7a84d7f65eeb790750ef96faed724c1a347239a584ab6e8612b042f3"} Oct 30 13:08:25.586784 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:25.586723 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" podUID="c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7" containerName="payload-processing" containerID="cri-o://6a67241a5ad94f978767b599658d275d86c4b62389f96fd05d1c8461e3d90cba" gracePeriod=25 Oct 30 13:08:25.802186 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:25.802162 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c99c3aa-rt7jk" Oct 30 13:08:25.865719 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:25.865659 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80712f3d-cc77-4955-86ce-3c66d7aac999-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80712f3d-cc77-4955-86ce-3c66d7aac999-maas-api-xks-ca\") pod \"80712f3d-cc77-4955-86ce-3c66d7aac999\" (UID: \"80712f3d-cc77-4955-86ce-3c66d7aac999\") " Oct 30 13:08:25.865719 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:25.865691 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80712f3d-cc77-4955-86ce-3c66d7aac999-kube-api-access-q2tn6\" (UniqueName: \"kubernetes.io/projected/80712f3d-cc77-4955-86ce-3c66d7aac999-kube-api-access-q2tn6\") pod \"80712f3d-cc77-4955-86ce-3c66d7aac999\" (UID: \"80712f3d-cc77-4955-86ce-3c66d7aac999\") " Oct 30 13:08:25.865929 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:25.865825 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80712f3d-cc77-4955-86ce-3c66d7aac999-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80712f3d-cc77-4955-86ce-3c66d7aac999-maas-api-openshift-ca\") pod \"80712f3d-cc77-4955-86ce-3c66d7aac999\" (UID: \"80712f3d-cc77-4955-86ce-3c66d7aac999\") " Oct 30 13:08:25.866256 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:25.866226 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80712f3d-cc77-4955-86ce-3c66d7aac999-maas-api-openshift-ca" pod "80712f3d-cc77-4955-86ce-3c66d7aac999" (UID: "80712f3d-cc77-4955-86ce-3c66d7aac999"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:25.867773 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:25.867748 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80712f3d-cc77-4955-86ce-3c66d7aac999-maas-api-xks-ca" pod "80712f3d-cc77-4955-86ce-3c66d7aac999" (UID: "80712f3d-cc77-4955-86ce-3c66d7aac999"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:25.867925 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:25.867863 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80712f3d-cc77-4955-86ce-3c66d7aac999-kube-api-access-q2tn6" pod "80712f3d-cc77-4955-86ce-3c66d7aac999" (UID: "80712f3d-cc77-4955-86ce-3c66d7aac999"). InnerVolumeSpecName "kube-api-access-q2tn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:25.967142 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:25.967116 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80712f3d-cc77-4955-86ce-3c66d7aac999-maas-api-openshift-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:25.967142 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:25.967142 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80712f3d-cc77-4955-86ce-3c66d7aac999-maas-api-xks-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:25.967285 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:25.967153 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2tn6\" (UniqueName: \"kubernetes.io/projected/80712f3d-cc77-4955-86ce-3c66d7aac999-kube-api-access-q2tn6\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:26.133907 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.133882 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 13:08:26.168496 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.168468 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-workload-socket\") pod \"4027e387-9043-4303-a6e1-c18365dcfac0\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " Oct 30 13:08:26.168637 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.168509 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-istio-data\") pod \"4027e387-9043-4303-a6e1-c18365dcfac0\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " Oct 30 13:08:26.168637 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.168598 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-istio-envoy\") pod \"4027e387-9043-4303-a6e1-c18365dcfac0\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " Oct 30 13:08:26.168757 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.168644 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-credential-socket\") pod \"4027e387-9043-4303-a6e1-c18365dcfac0\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " Oct 30 13:08:26.168757 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.168708 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4027e387-9043-4303-a6e1-c18365dcfac0-istio-token\" (UniqueName: \"kubernetes.io/projected/4027e387-9043-4303-a6e1-c18365dcfac0-istio-token\") pod \"4027e387-9043-4303-a6e1-c18365dcfac0\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " Oct 30 13:08:26.168757 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.168718 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-workload-socket" pod "4027e387-9043-4303-a6e1-c18365dcfac0" (UID: "4027e387-9043-4303-a6e1-c18365dcfac0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:26.168936 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.168744 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4027e387-9043-4303-a6e1-c18365dcfac0-kube-api-access-4nqtd\" (UniqueName: \"kubernetes.io/projected/4027e387-9043-4303-a6e1-c18365dcfac0-kube-api-access-4nqtd\") pod \"4027e387-9043-4303-a6e1-c18365dcfac0\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " Oct 30 13:08:26.168936 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.168792 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-istio-data" pod "4027e387-9043-4303-a6e1-c18365dcfac0" (UID: "4027e387-9043-4303-a6e1-c18365dcfac0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:26.168936 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.168844 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4027e387-9043-4303-a6e1-c18365dcfac0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4027e387-9043-4303-a6e1-c18365dcfac0-istio-podinfo\") pod \"4027e387-9043-4303-a6e1-c18365dcfac0\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " Oct 30 13:08:26.168936 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.168905 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4027e387-9043-4303-a6e1-c18365dcfac0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4027e387-9043-4303-a6e1-c18365dcfac0-istiod-ca-cert\") pod \"4027e387-9043-4303-a6e1-c18365dcfac0\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " Oct 30 13:08:26.169224 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.168942 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-workload-certs\") pod \"4027e387-9043-4303-a6e1-c18365dcfac0\" (UID: \"4027e387-9043-4303-a6e1-c18365dcfac0\") " Oct 30 13:08:26.169224 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.168987 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-credential-socket" pod "4027e387-9043-4303-a6e1-c18365dcfac0" (UID: "4027e387-9043-4303-a6e1-c18365dcfac0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:26.169345 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.169290 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-workload-certs" pod "4027e387-9043-4303-a6e1-c18365dcfac0" (UID: "4027e387-9043-4303-a6e1-c18365dcfac0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:26.169424 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.169391 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4027e387-9043-4303-a6e1-c18365dcfac0-istiod-ca-cert" pod "4027e387-9043-4303-a6e1-c18365dcfac0" (UID: "4027e387-9043-4303-a6e1-c18365dcfac0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:26.169486 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.169420 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:26.169486 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.169441 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:26.169486 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.169455 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:26.169486 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.169471 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:26.170918 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.170888 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4027e387-9043-4303-a6e1-c18365dcfac0-istio-podinfo" pod "4027e387-9043-4303-a6e1-c18365dcfac0" (UID: "4027e387-9043-4303-a6e1-c18365dcfac0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:08:26.170918 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.170910 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-istio-envoy" pod "4027e387-9043-4303-a6e1-c18365dcfac0" (UID: "4027e387-9043-4303-a6e1-c18365dcfac0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:26.171059 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.170934 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4027e387-9043-4303-a6e1-c18365dcfac0-istio-token" pod "4027e387-9043-4303-a6e1-c18365dcfac0" (UID: "4027e387-9043-4303-a6e1-c18365dcfac0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:26.171059 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.170958 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4027e387-9043-4303-a6e1-c18365dcfac0-kube-api-access-4nqtd" pod "4027e387-9043-4303-a6e1-c18365dcfac0" (UID: "4027e387-9043-4303-a6e1-c18365dcfac0"). InnerVolumeSpecName "kube-api-access-4nqtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:26.270928 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.270902 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4027e387-9043-4303-a6e1-c18365dcfac0-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:26.270928 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.270926 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4027e387-9043-4303-a6e1-c18365dcfac0-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:26.271137 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.270935 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nqtd\" (UniqueName: \"kubernetes.io/projected/4027e387-9043-4303-a6e1-c18365dcfac0-kube-api-access-4nqtd\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:26.271137 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.270947 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4027e387-9043-4303-a6e1-c18365dcfac0-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:26.271137 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.270956 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4027e387-9043-4303-a6e1-c18365dcfac0-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:26.583669 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.583625 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" podUID="4572318d-2ff5-43d0-9071-faeb6e94c178" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:26.583864 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.583631 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" podUID="127e9882-c84b-4162-9565-834a8126f9e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:26.648166 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.648128 4468 generic.go:356] "Generic (PLEG): container finished" podID="4027e387-9043-4303-a6e1-c18365dcfac0" containerID="dd308d1b1bbc57bbb94ec515a3ecdb0ef44846bd3f779e1b52da259b1970a841" exitCode=0 Oct 30 13:08:26.648312 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.648190 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" Oct 30 13:08:26.648312 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.648187 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" event={"ID":"4027e387-9043-4303-a6e1-c18365dcfac0","Type":"ContainerDied","Data":"dd308d1b1bbc57bbb94ec515a3ecdb0ef44846bd3f779e1b52da259b1970a841"} Oct 30 13:08:26.648312 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.648260 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s" event={"ID":"4027e387-9043-4303-a6e1-c18365dcfac0","Type":"ContainerDied","Data":"e108e4a735b3143a5a3fb9fa29add0ef0a87a62a26b1429bddea9577ee25e41f"} Oct 30 13:08:26.648312 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.648291 4468 scope.go:122] "RemoveContainer" containerID="dd308d1b1bbc57bbb94ec515a3ecdb0ef44846bd3f779e1b52da259b1970a841" Oct 30 13:08:26.649715 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.649682 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c99c3aa-rt7jk" event={"ID":"80712f3d-cc77-4955-86ce-3c66d7aac999","Type":"ContainerDied","Data":"35d99edd7a84d7f65eeb790750ef96faed724c1a347239a584ab6e8612b042f3"} Oct 30 13:08:26.649778 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.649720 4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35d99edd7a84d7f65eeb790750ef96faed724c1a347239a584ab6e8612b042f3" Oct 30 13:08:26.649829 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.649781 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c99c3aa-rt7jk" Oct 30 13:08:26.661485 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.661456 4468 scope.go:122] "RemoveContainer" containerID="7ca63977ebf67617613e536e0fade02abaa1701a88ea870a25bc7000feb0d398" Oct 30 13:08:26.677059 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.677038 4468 scope.go:122] "RemoveContainer" containerID="dd308d1b1bbc57bbb94ec515a3ecdb0ef44846bd3f779e1b52da259b1970a841" Oct 30 13:08:26.677336 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:08:26.677317 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd308d1b1bbc57bbb94ec515a3ecdb0ef44846bd3f779e1b52da259b1970a841\": container with ID starting with dd308d1b1bbc57bbb94ec515a3ecdb0ef44846bd3f779e1b52da259b1970a841 not found: ID does not exist" containerID="dd308d1b1bbc57bbb94ec515a3ecdb0ef44846bd3f779e1b52da259b1970a841" Oct 30 13:08:26.677396 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.677343 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd308d1b1bbc57bbb94ec515a3ecdb0ef44846bd3f779e1b52da259b1970a841"} err="failed to get container status \"dd308d1b1bbc57bbb94ec515a3ecdb0ef44846bd3f779e1b52da259b1970a841\": rpc error: code = NotFound desc = could not find container \"dd308d1b1bbc57bbb94ec515a3ecdb0ef44846bd3f779e1b52da259b1970a841\": container with ID starting with dd308d1b1bbc57bbb94ec515a3ecdb0ef44846bd3f779e1b52da259b1970a841 not found: ID does not exist" Oct 30 13:08:26.677396 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.677359 4468 scope.go:122] "RemoveContainer" containerID="7ca63977ebf67617613e536e0fade02abaa1701a88ea870a25bc7000feb0d398" Oct 30 13:08:26.677584 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:08:26.677569 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca63977ebf67617613e536e0fade02abaa1701a88ea870a25bc7000feb0d398\": container with ID starting with 7ca63977ebf67617613e536e0fade02abaa1701a88ea870a25bc7000feb0d398 not found: ID does not exist" containerID="7ca63977ebf67617613e536e0fade02abaa1701a88ea870a25bc7000feb0d398" Oct 30 13:08:26.677628 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.677589 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca63977ebf67617613e536e0fade02abaa1701a88ea870a25bc7000feb0d398"} err="failed to get container status \"7ca63977ebf67617613e536e0fade02abaa1701a88ea870a25bc7000feb0d398\": rpc error: code = NotFound desc = could not find container \"7ca63977ebf67617613e536e0fade02abaa1701a88ea870a25bc7000feb0d398\": container with ID starting with 7ca63977ebf67617613e536e0fade02abaa1701a88ea870a25bc7000feb0d398 not found: ID does not exist" Oct 30 13:08:26.722934 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.722909 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s"] Oct 30 13:08:26.735682 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:26.735659 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-327e31-openshift-default-c67659b97-7fs5s"] Oct 30 13:08:27.088798 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:27.088772 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4027e387-9043-4303-a6e1-c18365dcfac0" path="/var/lib/kubelet/pods/4027e387-9043-4303-a6e1-c18365dcfac0/volumes" Oct 30 13:08:29.505659 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:29.505620 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" podUID="c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:32.057439 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:32.057400 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5"] Oct 30 13:08:32.090417 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:32.090389 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz"] Oct 30 13:08:32.897701 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:32.897632 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz"] Oct 30 13:08:32.897941 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:32.897913 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" podUID="29260bd5-b3f6-4df0-8917-573986f5969a" containerName="maas-api" containerID="cri-o://34688e5e1f01d298f22e7078a90f26c76046d19c244e750653625484e9d2f78e" gracePeriod=30 Oct 30 13:08:33.157334 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.157314 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" Oct 30 13:08:33.244906 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.244869 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29260bd5-b3f6-4df0-8917-573986f5969a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29260bd5-b3f6-4df0-8917-573986f5969a-metrics-certs\") pod \"29260bd5-b3f6-4df0-8917-573986f5969a\" (UID: \"29260bd5-b3f6-4df0-8917-573986f5969a\") " Oct 30 13:08:33.245124 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.244927 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29260bd5-b3f6-4df0-8917-573986f5969a-kube-api-access-dd52c\" (UniqueName: \"kubernetes.io/projected/29260bd5-b3f6-4df0-8917-573986f5969a-kube-api-access-dd52c\") pod \"29260bd5-b3f6-4df0-8917-573986f5969a\" (UID: \"29260bd5-b3f6-4df0-8917-573986f5969a\") " Oct 30 13:08:33.245124 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.245011 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29260bd5-b3f6-4df0-8917-573986f5969a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29260bd5-b3f6-4df0-8917-573986f5969a-maas-api-tls\") pod \"29260bd5-b3f6-4df0-8917-573986f5969a\" (UID: \"29260bd5-b3f6-4df0-8917-573986f5969a\") " Oct 30 13:08:33.245124 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.245038 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29260bd5-b3f6-4df0-8917-573986f5969a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29260bd5-b3f6-4df0-8917-573986f5969a-openshift-service-ca\") pod \"29260bd5-b3f6-4df0-8917-573986f5969a\" (UID: \"29260bd5-b3f6-4df0-8917-573986f5969a\") " Oct 30 13:08:33.245528 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.245493 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29260bd5-b3f6-4df0-8917-573986f5969a-openshift-service-ca" pod "29260bd5-b3f6-4df0-8917-573986f5969a" (UID: "29260bd5-b3f6-4df0-8917-573986f5969a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:33.247316 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.247283 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29260bd5-b3f6-4df0-8917-573986f5969a-maas-api-tls" pod "29260bd5-b3f6-4df0-8917-573986f5969a" (UID: "29260bd5-b3f6-4df0-8917-573986f5969a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:33.247406 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.247315 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29260bd5-b3f6-4df0-8917-573986f5969a-metrics-certs" pod "29260bd5-b3f6-4df0-8917-573986f5969a" (UID: "29260bd5-b3f6-4df0-8917-573986f5969a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:33.247406 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.247347 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29260bd5-b3f6-4df0-8917-573986f5969a-kube-api-access-dd52c" pod "29260bd5-b3f6-4df0-8917-573986f5969a" (UID: "29260bd5-b3f6-4df0-8917-573986f5969a"). InnerVolumeSpecName "kube-api-access-dd52c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:33.346480 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.346445 4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29260bd5-b3f6-4df0-8917-573986f5969a-metrics-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:33.346654 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.346498 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd52c\" (UniqueName: \"kubernetes.io/projected/29260bd5-b3f6-4df0-8917-573986f5969a-kube-api-access-dd52c\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:33.346654 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.346509 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29260bd5-b3f6-4df0-8917-573986f5969a-maas-api-tls\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:33.346654 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.346518 4468 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29260bd5-b3f6-4df0-8917-573986f5969a-openshift-service-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:33.687879 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.687845 4468 generic.go:356] "Generic (PLEG): container finished" podID="29260bd5-b3f6-4df0-8917-573986f5969a" containerID="34688e5e1f01d298f22e7078a90f26c76046d19c244e750653625484e9d2f78e" exitCode=0 Oct 30 13:08:33.688119 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.687897 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" Oct 30 13:08:33.688119 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.687924 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" event={"ID":"29260bd5-b3f6-4df0-8917-573986f5969a","Type":"ContainerDied","Data":"34688e5e1f01d298f22e7078a90f26c76046d19c244e750653625484e9d2f78e"} Oct 30 13:08:33.688119 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.687983 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz" event={"ID":"29260bd5-b3f6-4df0-8917-573986f5969a","Type":"ContainerDied","Data":"2c604f45dcf744148703fc0160c8928cb5456a6c08ca8a6d0d1b3928a8f12a0e"} Oct 30 13:08:33.688119 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.688007 4468 scope.go:122] "RemoveContainer" containerID="34688e5e1f01d298f22e7078a90f26c76046d19c244e750653625484e9d2f78e" Oct 30 13:08:33.698671 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.698640 4468 scope.go:122] "RemoveContainer" containerID="34688e5e1f01d298f22e7078a90f26c76046d19c244e750653625484e9d2f78e" Oct 30 13:08:33.698959 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:08:33.698933 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34688e5e1f01d298f22e7078a90f26c76046d19c244e750653625484e9d2f78e\": container with ID starting with 34688e5e1f01d298f22e7078a90f26c76046d19c244e750653625484e9d2f78e not found: ID does not exist" containerID="34688e5e1f01d298f22e7078a90f26c76046d19c244e750653625484e9d2f78e" Oct 30 13:08:33.699107 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.698966 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34688e5e1f01d298f22e7078a90f26c76046d19c244e750653625484e9d2f78e"} err="failed to get container status \"34688e5e1f01d298f22e7078a90f26c76046d19c244e750653625484e9d2f78e\": rpc error: code = NotFound desc = could not find container \"34688e5e1f01d298f22e7078a90f26c76046d19c244e750653625484e9d2f78e\": container with ID starting with 34688e5e1f01d298f22e7078a90f26c76046d19c244e750653625484e9d2f78e not found: ID does not exist" Oct 30 13:08:33.760496 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.760464 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz"] Oct 30 13:08:33.773961 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:33.773930 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-8d2170-644787b598-tkbzz"] Oct 30 13:08:35.087867 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:35.087825 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29260bd5-b3f6-4df0-8917-573986f5969a" path="/var/lib/kubelet/pods/29260bd5-b3f6-4df0-8917-573986f5969a/volumes" Oct 30 13:08:36.583356 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:36.583313 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" podUID="4572318d-2ff5-43d0-9071-faeb6e94c178" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:36.583356 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:36.583326 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" podUID="127e9882-c84b-4162-9565-834a8126f9e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:37.058399 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.058353 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" podUID="0022531e-0b86-46d8-b59e-ed23010efa07" containerName="payload-processing" containerID="cri-o://69c35757960310240363fde0857945765d07e5bfa89603294c2f3ed3f100cc7a" gracePeriod=25 Oct 30 13:08:37.091141 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.091111 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" podUID="179f5f3b-6ed4-4330-91ee-6dcbf00ea87a" containerName="payload-pre-processing" containerID="cri-o://8f5cde7c40b46ed60cf63473b3f046c56748a71abbd67c87cff7fea392e06d5b" gracePeriod=25 Oct 30 13:08:37.098359 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.098339 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" Oct 30 13:08:37.119201 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.119180 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" Oct 30 13:08:37.183052 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.183024 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/127e9882-c84b-4162-9565-834a8126f9e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127e9882-c84b-4162-9565-834a8126f9e3-plugins-config-volume\") pod \"127e9882-c84b-4162-9565-834a8126f9e3\" (UID: \"127e9882-c84b-4162-9565-834a8126f9e3\") " Oct 30 13:08:37.183223 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.183098 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4572318d-2ff5-43d0-9071-faeb6e94c178-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4572318d-2ff5-43d0-9071-faeb6e94c178-plugins-config-volume\") pod \"4572318d-2ff5-43d0-9071-faeb6e94c178\" (UID: \"4572318d-2ff5-43d0-9071-faeb6e94c178\") " Oct 30 13:08:37.183223 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.183177 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4572318d-2ff5-43d0-9071-faeb6e94c178-kube-api-access-xwwnk\" (UniqueName: \"kubernetes.io/projected/4572318d-2ff5-43d0-9071-faeb6e94c178-kube-api-access-xwwnk\") pod \"4572318d-2ff5-43d0-9071-faeb6e94c178\" (UID: \"4572318d-2ff5-43d0-9071-faeb6e94c178\") " Oct 30 13:08:37.183345 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.183260 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/127e9882-c84b-4162-9565-834a8126f9e3-kube-api-access-r6mzr\" (UniqueName: \"kubernetes.io/projected/127e9882-c84b-4162-9565-834a8126f9e3-kube-api-access-r6mzr\") pod \"127e9882-c84b-4162-9565-834a8126f9e3\" (UID: \"127e9882-c84b-4162-9565-834a8126f9e3\") " Oct 30 13:08:37.183500 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.183474 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/127e9882-c84b-4162-9565-834a8126f9e3-plugins-config-volume" pod "127e9882-c84b-4162-9565-834a8126f9e3" (UID: "127e9882-c84b-4162-9565-834a8126f9e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:37.183578 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.183522 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4572318d-2ff5-43d0-9071-faeb6e94c178-plugins-config-volume" pod "4572318d-2ff5-43d0-9071-faeb6e94c178" (UID: "4572318d-2ff5-43d0-9071-faeb6e94c178"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:37.183663 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.183645 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127e9882-c84b-4162-9565-834a8126f9e3-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:37.183721 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.183665 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4572318d-2ff5-43d0-9071-faeb6e94c178-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:37.185605 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.185566 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127e9882-c84b-4162-9565-834a8126f9e3-kube-api-access-r6mzr" pod "127e9882-c84b-4162-9565-834a8126f9e3" (UID: "127e9882-c84b-4162-9565-834a8126f9e3"). InnerVolumeSpecName "kube-api-access-r6mzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:37.185740 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.185719 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4572318d-2ff5-43d0-9071-faeb6e94c178-kube-api-access-xwwnk" pod "4572318d-2ff5-43d0-9071-faeb6e94c178" (UID: "4572318d-2ff5-43d0-9071-faeb6e94c178"). InnerVolumeSpecName "kube-api-access-xwwnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:37.284979 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.284952 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwwnk\" (UniqueName: \"kubernetes.io/projected/4572318d-2ff5-43d0-9071-faeb6e94c178-kube-api-access-xwwnk\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:37.284979 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.284975 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6mzr\" (UniqueName: \"kubernetes.io/projected/127e9882-c84b-4162-9565-834a8126f9e3-kube-api-access-r6mzr\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:37.710399 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.710362 4468 generic.go:356] "Generic (PLEG): container finished" podID="4572318d-2ff5-43d0-9071-faeb6e94c178" containerID="707155a85afee81f8881e08418e961fccbb2123b7f25ff1b8d0e34d273172f56" exitCode=0 Oct 30 13:08:37.710823 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.710417 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" Oct 30 13:08:37.710823 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.710442 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" event={"ID":"4572318d-2ff5-43d0-9071-faeb6e94c178","Type":"ContainerDied","Data":"707155a85afee81f8881e08418e961fccbb2123b7f25ff1b8d0e34d273172f56"} Oct 30 13:08:37.710823 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.710501 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn" event={"ID":"4572318d-2ff5-43d0-9071-faeb6e94c178","Type":"ContainerDied","Data":"87ff20470cb3380054592a2b8b9a4e81e7b7acdb161c3b4bc1e016f132c7175b"} Oct 30 13:08:37.710823 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.710525 4468 scope.go:122] "RemoveContainer" containerID="707155a85afee81f8881e08418e961fccbb2123b7f25ff1b8d0e34d273172f56" Oct 30 13:08:37.712086 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.712044 4468 generic.go:356] "Generic (PLEG): container finished" podID="127e9882-c84b-4162-9565-834a8126f9e3" containerID="645e122f975e1e42c9c9766f80a4b290047cb9b7e75429a172953a51a91663a8" exitCode=0 Oct 30 13:08:37.712200 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.712095 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" Oct 30 13:08:37.712200 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.712110 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" event={"ID":"127e9882-c84b-4162-9565-834a8126f9e3","Type":"ContainerDied","Data":"645e122f975e1e42c9c9766f80a4b290047cb9b7e75429a172953a51a91663a8"} Oct 30 13:08:37.712200 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.712176 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s" event={"ID":"127e9882-c84b-4162-9565-834a8126f9e3","Type":"ContainerDied","Data":"f20153659601b8a9f0d332953bcf9e7184688985211d041e4fdb8c09535cfbb3"} Oct 30 13:08:37.719862 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.719844 4468 scope.go:122] "RemoveContainer" containerID="707155a85afee81f8881e08418e961fccbb2123b7f25ff1b8d0e34d273172f56" Oct 30 13:08:37.720133 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:08:37.720114 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"707155a85afee81f8881e08418e961fccbb2123b7f25ff1b8d0e34d273172f56\": container with ID starting with 707155a85afee81f8881e08418e961fccbb2123b7f25ff1b8d0e34d273172f56 not found: ID does not exist" containerID="707155a85afee81f8881e08418e961fccbb2123b7f25ff1b8d0e34d273172f56" Oct 30 13:08:37.720205 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.720139 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"707155a85afee81f8881e08418e961fccbb2123b7f25ff1b8d0e34d273172f56"} err="failed to get container status \"707155a85afee81f8881e08418e961fccbb2123b7f25ff1b8d0e34d273172f56\": rpc error: code = NotFound desc = could not find container \"707155a85afee81f8881e08418e961fccbb2123b7f25ff1b8d0e34d273172f56\": container with ID starting with 707155a85afee81f8881e08418e961fccbb2123b7f25ff1b8d0e34d273172f56 not found: ID does not exist" Oct 30 13:08:37.720205 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.720155 4468 scope.go:122] "RemoveContainer" containerID="645e122f975e1e42c9c9766f80a4b290047cb9b7e75429a172953a51a91663a8" Oct 30 13:08:37.728013 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.727994 4468 scope.go:122] "RemoveContainer" containerID="645e122f975e1e42c9c9766f80a4b290047cb9b7e75429a172953a51a91663a8" Oct 30 13:08:37.728349 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:08:37.728329 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"645e122f975e1e42c9c9766f80a4b290047cb9b7e75429a172953a51a91663a8\": container with ID starting with 645e122f975e1e42c9c9766f80a4b290047cb9b7e75429a172953a51a91663a8 not found: ID does not exist" containerID="645e122f975e1e42c9c9766f80a4b290047cb9b7e75429a172953a51a91663a8" Oct 30 13:08:37.728428 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.728354 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"645e122f975e1e42c9c9766f80a4b290047cb9b7e75429a172953a51a91663a8"} err="failed to get container status \"645e122f975e1e42c9c9766f80a4b290047cb9b7e75429a172953a51a91663a8\": rpc error: code = NotFound desc = could not find container \"645e122f975e1e42c9c9766f80a4b290047cb9b7e75429a172953a51a91663a8\": container with ID starting with 645e122f975e1e42c9c9766f80a4b290047cb9b7e75429a172953a51a91663a8 not found: ID does not exist" Oct 30 13:08:37.767835 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.767813 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn"] Oct 30 13:08:37.770549 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.770528 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-47fd99-8688dd8dd8-r27mn"] Oct 30 13:08:37.809922 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.809901 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s"] Oct 30 13:08:37.816325 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:37.816301 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-47fd99-7b57d497d7-8tx7s"] Oct 30 13:08:39.090284 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:39.090252 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="127e9882-c84b-4162-9565-834a8126f9e3" path="/var/lib/kubelet/pods/127e9882-c84b-4162-9565-834a8126f9e3/volumes" Oct 30 13:08:39.090787 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:39.090772 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4572318d-2ff5-43d0-9071-faeb6e94c178" path="/var/lib/kubelet/pods/4572318d-2ff5-43d0-9071-faeb6e94c178/volumes" Oct 30 13:08:39.505707 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:39.505668 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" podUID="c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:41.137512 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:41.137475 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" podUID="0022531e-0b86-46d8-b59e-ed23010efa07" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:42.087854 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.087814 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn"] Oct 30 13:08:42.088332 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.088313 4468 cpu_manager.go:414] "containerMap: removing container" podUID="80712f3d-cc77-4955-86ce-3c66d7aac999" containerName="revoke-keys" Oct 30 13:08:42.088442 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.088333 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80712f3d-cc77-4955-86ce-3c66d7aac999" containerName="revoke-keys" Oct 30 13:08:42.088442 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.088351 4468 cpu_manager.go:414] "containerMap: removing container" podUID="4027e387-9043-4303-a6e1-c18365dcfac0" containerName="istio-proxy" Oct 30 13:08:42.088442 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.088359 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4027e387-9043-4303-a6e1-c18365dcfac0" containerName="istio-proxy" Oct 30 13:08:42.088442 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.088391 4468 cpu_manager.go:414] "containerMap: removing container" podUID="29260bd5-b3f6-4df0-8917-573986f5969a" containerName="maas-api" Oct 30 13:08:42.088442 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.088399 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29260bd5-b3f6-4df0-8917-573986f5969a" containerName="maas-api" Oct 30 13:08:42.088442 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.088412 4468 cpu_manager.go:414] "containerMap: removing container" podUID="127e9882-c84b-4162-9565-834a8126f9e3" containerName="payload-pre-processing" Oct 30 13:08:42.088442 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.088419 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="127e9882-c84b-4162-9565-834a8126f9e3" containerName="payload-pre-processing" Oct 30 13:08:42.088810 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.088447 4468 cpu_manager.go:414] "containerMap: removing container" podUID="4572318d-2ff5-43d0-9071-faeb6e94c178" containerName="payload-processing" Oct 30 13:08:42.088810 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.088457 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4572318d-2ff5-43d0-9071-faeb6e94c178" containerName="payload-processing" Oct 30 13:08:42.088810 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.088558 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="29260bd5-b3f6-4df0-8917-573986f5969a" containerName="maas-api" Oct 30 13:08:42.088810 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.088574 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="127e9882-c84b-4162-9565-834a8126f9e3" containerName="payload-pre-processing" Oct 30 13:08:42.088810 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.088584 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4027e387-9043-4303-a6e1-c18365dcfac0" containerName="istio-proxy" Oct 30 13:08:42.088810 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.088597 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4572318d-2ff5-43d0-9071-faeb6e94c178" containerName="payload-processing" Oct 30 13:08:42.088810 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.088607 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="80712f3d-cc77-4955-86ce-3c66d7aac999" containerName="revoke-keys" Oct 30 13:08:42.088810 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.088616 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4027e387-9043-4303-a6e1-c18365dcfac0" containerName="istio-proxy" Oct 30 13:08:42.091377 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.091357 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn" Oct 30 13:08:42.112500 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.112475 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn"] Oct 30 13:08:42.120572 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.120550 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:08:42.120668 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.120597 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:08:42.146433 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.146409 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" podUID="179f5f3b-6ed4-4330-91ee-6dcbf00ea87a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:42.230520 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.230488 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn\" (UID: \"a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn" Oct 30 13:08:42.230634 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.230528 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx88m\" (UniqueName: \"kubernetes.io/projected/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-kube-api-access-qx88m\") pod \"maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn\" (UID: \"a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn" Oct 30 13:08:42.230749 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.230724 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn\" (UID: \"a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn" Oct 30 13:08:42.331752 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.331719 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn\" (UID: \"a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn" Oct 30 13:08:42.331875 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.331800 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn\" (UID: \"a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn" Oct 30 13:08:42.331875 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.331844 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx88m\" (UniqueName: \"kubernetes.io/projected/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-kube-api-access-qx88m\") pod \"maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn\" (UID: \"a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn" Oct 30 13:08:42.332480 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.332459 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn\" (UID: \"a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn" Oct 30 13:08:42.334142 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.334121 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn\" (UID: \"a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn" Oct 30 13:08:42.356030 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.355980 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx88m\" (UniqueName: \"kubernetes.io/projected/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-kube-api-access-qx88m\") pod \"maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn\" (UID: \"a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn" Oct 30 13:08:42.402974 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.402947 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn" Oct 30 13:08:42.716390 ip-10-0-131-76 kubenswrapper[4468]: W1030 13:08:42.716357 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a3bc74_bfc3_4ccb_ab1b_99dc6f3c0db8.slice/crio-da4d78189dabcb41c1796240c2410b3de5e5561909196fc53f5b6609f56ab6fd WatchSource:0}: Error finding container da4d78189dabcb41c1796240c2410b3de5e5561909196fc53f5b6609f56ab6fd: Status 404 returned error can't find the container with id da4d78189dabcb41c1796240c2410b3de5e5561909196fc53f5b6609f56ab6fd Oct 30 13:08:42.719065 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.719037 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn"] Oct 30 13:08:42.738737 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:42.738713 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn" event={"ID":"a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8","Type":"ContainerStarted","Data":"da4d78189dabcb41c1796240c2410b3de5e5561909196fc53f5b6609f56ab6fd"} Oct 30 13:08:43.746125 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:43.746064 4468 generic.go:356] "Generic (PLEG): container finished" podID="a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8" containerID="e1407872014e28919f841ee511358c345e8080085d8b133dc7499fe055097aa8" exitCode=0 Oct 30 13:08:43.746484 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:43.746160 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn" event={"ID":"a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8","Type":"ContainerDied","Data":"e1407872014e28919f841ee511358c345e8080085d8b133dc7499fe055097aa8"} Oct 30 13:08:44.887367 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:44.887341 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn" Oct 30 13:08:44.958127 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:44.958102 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-maas-api-xks-ca\") pod \"a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8\" (UID: \"a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8\") " Oct 30 13:08:44.958249 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:44.958189 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-kube-api-access-qx88m\" (UniqueName: \"kubernetes.io/projected/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-kube-api-access-qx88m\") pod \"a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8\" (UID: \"a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8\") " Oct 30 13:08:44.958293 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:44.958259 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-maas-api-openshift-ca\") pod \"a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8\" (UID: \"a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8\") " Oct 30 13:08:44.958655 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:44.958626 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-maas-api-openshift-ca" pod "a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8" (UID: "a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:44.960112 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:44.960078 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-maas-api-xks-ca" pod "a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8" (UID: "a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:44.960255 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:44.960149 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-kube-api-access-qx88m" pod "a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8" (UID: "a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8"). InnerVolumeSpecName "kube-api-access-qx88m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:45.059467 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:45.059413 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx88m\" (UniqueName: \"kubernetes.io/projected/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-kube-api-access-qx88m\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:45.059467 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:45.059436 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-maas-api-openshift-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:45.059467 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:45.059446 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8-maas-api-xks-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:45.761193 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:45.760817 4468 generic.go:356] "Generic (PLEG): container finished" podID="c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7" containerID="6a67241a5ad94f978767b599658d275d86c4b62389f96fd05d1c8461e3d90cba" exitCode=0 Oct 30 13:08:45.761193 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:45.760870 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" event={"ID":"c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7","Type":"ContainerDied","Data":"6a67241a5ad94f978767b599658d275d86c4b62389f96fd05d1c8461e3d90cba"} Oct 30 13:08:45.762601 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:45.762577 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn" Oct 30 13:08:45.762722 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:45.762583 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn" event={"ID":"a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8","Type":"ContainerDied","Data":"da4d78189dabcb41c1796240c2410b3de5e5561909196fc53f5b6609f56ab6fd"} Oct 30 13:08:45.762722 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:45.762694 4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da4d78189dabcb41c1796240c2410b3de5e5561909196fc53f5b6609f56ab6fd" Oct 30 13:08:45.821877 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:45.821854 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" Oct 30 13:08:45.967282 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:45.967251 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7-kube-api-access-br9v8\" (UniqueName: \"kubernetes.io/projected/c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7-kube-api-access-br9v8\") pod \"c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7\" (UID: \"c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7\") " Oct 30 13:08:45.967734 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:45.967317 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7-plugins-config-volume\") pod \"c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7\" (UID: \"c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7\") " Oct 30 13:08:45.967734 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:45.967672 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7-plugins-config-volume" pod "c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7" (UID: "c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:45.969361 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:45.969341 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7-kube-api-access-br9v8" pod "c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7" (UID: "c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7"). InnerVolumeSpecName "kube-api-access-br9v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:46.068446 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:46.068419 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br9v8\" (UniqueName: \"kubernetes.io/projected/c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7-kube-api-access-br9v8\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:46.068446 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:46.068443 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:46.769142 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:46.769102 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" event={"ID":"c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7","Type":"ContainerDied","Data":"ac74747cb26c88fc9715a2ea61f2f5bc8fd7532012f3afbb9906066c5e138d68"} Oct 30 13:08:46.769345 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:46.769116 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7" Oct 30 13:08:46.769345 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:46.769186 4468 scope.go:122] "RemoveContainer" containerID="6a67241a5ad94f978767b599658d275d86c4b62389f96fd05d1c8461e3d90cba" Oct 30 13:08:46.817794 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:46.817758 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5"] Oct 30 13:08:46.827499 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:46.827480 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5"] Oct 30 13:08:46.834444 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:46.834424 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7"] Oct 30 13:08:46.842290 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:46.842266 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6a7148-7b89464846-hktb7"] Oct 30 13:08:47.087450 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:47.087382 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7" path="/var/lib/kubelet/pods/c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7/volumes" Oct 30 13:08:48.705325 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:48.705291 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt"] Oct 30 13:08:48.705743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:48.705522 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" podUID="ea87399a-8203-4239-a086-097a4bb9120e" containerName="istio-proxy" containerID="cri-o://faecc3dbf16cbbca0b9a82f68f05f67854150721c0835d14cbc0d240e04106a7" gracePeriod=30 Oct 30 13:08:51.137744 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:51.137706 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" podUID="0022531e-0b86-46d8-b59e-ed23010efa07" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:52.146157 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:52.146113 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" podUID="179f5f3b-6ed4-4330-91ee-6dcbf00ea87a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:52.742568 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:52.742534 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt"] Oct 30 13:08:52.801688 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:52.801653 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2"] Oct 30 13:08:53.598060 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:08:53.598019 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:08:53.958902 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:53.958879 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 13:08:54.042881 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.042847 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea87399a-8203-4239-a086-097a4bb9120e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea87399a-8203-4239-a086-097a4bb9120e-istio-podinfo\") pod \"ea87399a-8203-4239-a086-097a4bb9120e\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " Oct 30 13:08:54.043053 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.042937 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea87399a-8203-4239-a086-097a4bb9120e-istio-token\" (UniqueName: \"kubernetes.io/projected/ea87399a-8203-4239-a086-097a4bb9120e-istio-token\") pod \"ea87399a-8203-4239-a086-097a4bb9120e\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " Oct 30 13:08:54.043053 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.042978 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-istio-data\") pod \"ea87399a-8203-4239-a086-097a4bb9120e\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " Oct 30 13:08:54.043053 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.043030 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-istio-envoy\") pod \"ea87399a-8203-4239-a086-097a4bb9120e\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " Oct 30 13:08:54.043248 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.043064 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-credential-socket\") pod \"ea87399a-8203-4239-a086-097a4bb9120e\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " Oct 30 13:08:54.043248 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.043109 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-workload-socket\") pod \"ea87399a-8203-4239-a086-097a4bb9120e\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " Oct 30 13:08:54.043248 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.043142 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea87399a-8203-4239-a086-097a4bb9120e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea87399a-8203-4239-a086-097a4bb9120e-istiod-ca-cert\") pod \"ea87399a-8203-4239-a086-097a4bb9120e\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " Oct 30 13:08:54.043248 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.043169 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-workload-certs\") pod \"ea87399a-8203-4239-a086-097a4bb9120e\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " Oct 30 13:08:54.043248 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.043209 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea87399a-8203-4239-a086-097a4bb9120e-kube-api-access-jjxmn\" (UniqueName: \"kubernetes.io/projected/ea87399a-8203-4239-a086-097a4bb9120e-kube-api-access-jjxmn\") pod \"ea87399a-8203-4239-a086-097a4bb9120e\" (UID: \"ea87399a-8203-4239-a086-097a4bb9120e\") " Oct 30 13:08:54.043605 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.043385 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-istio-data" pod "ea87399a-8203-4239-a086-097a4bb9120e" (UID: "ea87399a-8203-4239-a086-097a4bb9120e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:54.043605 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.043591 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:54.043888 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.043842 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-workload-socket" pod "ea87399a-8203-4239-a086-097a4bb9120e" (UID: "ea87399a-8203-4239-a086-097a4bb9120e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:54.044095 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.044053 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-credential-socket" pod "ea87399a-8203-4239-a086-097a4bb9120e" (UID: "ea87399a-8203-4239-a086-097a4bb9120e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:54.044181 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.043993 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea87399a-8203-4239-a086-097a4bb9120e-istiod-ca-cert" pod "ea87399a-8203-4239-a086-097a4bb9120e" (UID: "ea87399a-8203-4239-a086-097a4bb9120e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:54.044285 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.044266 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-workload-certs" pod "ea87399a-8203-4239-a086-097a4bb9120e" (UID: "ea87399a-8203-4239-a086-097a4bb9120e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:54.045670 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.045632 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea87399a-8203-4239-a086-097a4bb9120e-kube-api-access-jjxmn" pod "ea87399a-8203-4239-a086-097a4bb9120e" (UID: "ea87399a-8203-4239-a086-097a4bb9120e"). InnerVolumeSpecName "kube-api-access-jjxmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:54.045891 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.045867 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea87399a-8203-4239-a086-097a4bb9120e-istio-podinfo" pod "ea87399a-8203-4239-a086-097a4bb9120e" (UID: "ea87399a-8203-4239-a086-097a4bb9120e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:08:54.046009 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.045906 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea87399a-8203-4239-a086-097a4bb9120e-istio-token" pod "ea87399a-8203-4239-a086-097a4bb9120e" (UID: "ea87399a-8203-4239-a086-097a4bb9120e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:54.046189 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.046171 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-istio-envoy" pod "ea87399a-8203-4239-a086-097a4bb9120e" (UID: "ea87399a-8203-4239-a086-097a4bb9120e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:54.144693 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.144642 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:54.144693 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.144664 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:54.144693 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.144674 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:54.144693 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.144683 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea87399a-8203-4239-a086-097a4bb9120e-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:54.144693 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.144691 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea87399a-8203-4239-a086-097a4bb9120e-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:54.144920 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.144699 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjxmn\" (UniqueName: \"kubernetes.io/projected/ea87399a-8203-4239-a086-097a4bb9120e-kube-api-access-jjxmn\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:54.144920 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.144707 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea87399a-8203-4239-a086-097a4bb9120e-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:54.144920 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.144715 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea87399a-8203-4239-a086-097a4bb9120e-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:54.812742 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.812706 4468 generic.go:356] "Generic (PLEG): container finished" podID="ea87399a-8203-4239-a086-097a4bb9120e" containerID="faecc3dbf16cbbca0b9a82f68f05f67854150721c0835d14cbc0d240e04106a7" exitCode=0 Oct 30 13:08:54.813262 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.812762 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" event={"ID":"ea87399a-8203-4239-a086-097a4bb9120e","Type":"ContainerDied","Data":"faecc3dbf16cbbca0b9a82f68f05f67854150721c0835d14cbc0d240e04106a7"} Oct 30 13:08:54.813262 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.812778 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" Oct 30 13:08:54.813262 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.812828 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt" event={"ID":"ea87399a-8203-4239-a086-097a4bb9120e","Type":"ContainerDied","Data":"262639102a7d38a4ecfde246480b5d98614c238887a6748e2aae9bc128730340"} Oct 30 13:08:54.813262 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.812851 4468 scope.go:122] "RemoveContainer" containerID="faecc3dbf16cbbca0b9a82f68f05f67854150721c0835d14cbc0d240e04106a7" Oct 30 13:08:54.822262 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.822245 4468 scope.go:122] "RemoveContainer" containerID="faecc3dbf16cbbca0b9a82f68f05f67854150721c0835d14cbc0d240e04106a7" Oct 30 13:08:54.822534 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:08:54.822516 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faecc3dbf16cbbca0b9a82f68f05f67854150721c0835d14cbc0d240e04106a7\": container with ID starting with faecc3dbf16cbbca0b9a82f68f05f67854150721c0835d14cbc0d240e04106a7 not found: ID does not exist" containerID="faecc3dbf16cbbca0b9a82f68f05f67854150721c0835d14cbc0d240e04106a7" Oct 30 13:08:54.822604 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.822543 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faecc3dbf16cbbca0b9a82f68f05f67854150721c0835d14cbc0d240e04106a7"} err="failed to get container status \"faecc3dbf16cbbca0b9a82f68f05f67854150721c0835d14cbc0d240e04106a7\": rpc error: code = NotFound desc = could not find container \"faecc3dbf16cbbca0b9a82f68f05f67854150721c0835d14cbc0d240e04106a7\": container with ID starting with faecc3dbf16cbbca0b9a82f68f05f67854150721c0835d14cbc0d240e04106a7 not found: ID does not exist" Oct 30 13:08:54.884213 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.884185 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt"] Oct 30 13:08:54.893025 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:54.892997 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-6dqgt"] Oct 30 13:08:55.088432 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.088374 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea87399a-8203-4239-a086-097a4bb9120e" path="/var/lib/kubelet/pods/ea87399a-8203-4239-a086-097a4bb9120e/volumes" Oct 30 13:08:55.719626 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.719591 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph"] Oct 30 13:08:55.720090 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.720057 4468 cpu_manager.go:414] "containerMap: removing container" podUID="4027e387-9043-4303-a6e1-c18365dcfac0" containerName="istio-proxy" Oct 30 13:08:55.720224 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.720091 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4027e387-9043-4303-a6e1-c18365dcfac0" containerName="istio-proxy" Oct 30 13:08:55.720224 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.720107 4468 cpu_manager.go:414] "containerMap: removing container" podUID="a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8" containerName="revoke-keys" Oct 30 13:08:55.720224 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.720115 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8" containerName="revoke-keys" Oct 30 13:08:55.720224 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.720140 4468 cpu_manager.go:414] "containerMap: removing container" podUID="c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7" containerName="payload-processing" Oct 30 13:08:55.720224 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.720150 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7" containerName="payload-processing" Oct 30 13:08:55.720224 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.720172 4468 cpu_manager.go:414] "containerMap: removing container" podUID="ea87399a-8203-4239-a086-097a4bb9120e" containerName="istio-proxy" Oct 30 13:08:55.720224 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.720179 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea87399a-8203-4239-a086-097a4bb9120e" containerName="istio-proxy" Oct 30 13:08:55.720772 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.720281 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d0e8b8-6a27-4d23-a6be-f1b8421a55c7" containerName="payload-processing" Oct 30 13:08:55.720772 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.720297 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea87399a-8203-4239-a086-097a4bb9120e" containerName="istio-proxy" Oct 30 13:08:55.720772 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.720313 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8" containerName="revoke-keys" Oct 30 13:08:55.724931 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.724910 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph" Oct 30 13:08:55.742365 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.742335 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph"] Oct 30 13:08:55.752405 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.752386 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:08:55.752529 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.752390 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:08:55.861983 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.861951 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de18bc3-57lph\" (UID: \"f3b0d918-b8bc-4367-9c8f-fe90da2e92be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph" Oct 30 13:08:55.862454 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.861997 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de18bc3-57lph\" (UID: \"f3b0d918-b8bc-4367-9c8f-fe90da2e92be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph" Oct 30 13:08:55.862454 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.862211 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmrdp\" (UniqueName: \"kubernetes.io/projected/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-kube-api-access-dmrdp\") pod \"maas-api-revoke-sub-2de18bc3-57lph\" (UID: \"f3b0d918-b8bc-4367-9c8f-fe90da2e92be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph" Oct 30 13:08:55.963362 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.963334 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de18bc3-57lph\" (UID: \"f3b0d918-b8bc-4367-9c8f-fe90da2e92be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph" Oct 30 13:08:55.963532 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.963375 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de18bc3-57lph\" (UID: \"f3b0d918-b8bc-4367-9c8f-fe90da2e92be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph" Oct 30 13:08:55.963532 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.963445 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmrdp\" (UniqueName: \"kubernetes.io/projected/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-kube-api-access-dmrdp\") pod \"maas-api-revoke-sub-2de18bc3-57lph\" (UID: \"f3b0d918-b8bc-4367-9c8f-fe90da2e92be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph" Oct 30 13:08:55.964056 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.964031 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de18bc3-57lph\" (UID: \"f3b0d918-b8bc-4367-9c8f-fe90da2e92be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph" Oct 30 13:08:55.965704 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.965682 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de18bc3-57lph\" (UID: \"f3b0d918-b8bc-4367-9c8f-fe90da2e92be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph" Oct 30 13:08:55.991115 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:55.991052 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmrdp\" (UniqueName: \"kubernetes.io/projected/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-kube-api-access-dmrdp\") pod \"maas-api-revoke-sub-2de18bc3-57lph\" (UID: \"f3b0d918-b8bc-4367-9c8f-fe90da2e92be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph" Oct 30 13:08:56.038184 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:56.038163 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph" Oct 30 13:08:56.342208 ip-10-0-131-76 kubenswrapper[4468]: W1030 13:08:56.342181 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b0d918_b8bc_4367_9c8f_fe90da2e92be.slice/crio-e149423d5818ba02adcbf42156f340b52d3789757bb8d743baed822f4100c9e0 WatchSource:0}: Error finding container e149423d5818ba02adcbf42156f340b52d3789757bb8d743baed822f4100c9e0: Status 404 returned error can't find the container with id e149423d5818ba02adcbf42156f340b52d3789757bb8d743baed822f4100c9e0 Oct 30 13:08:56.342348 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:56.342326 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph"] Oct 30 13:08:56.826660 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:56.826631 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph" event={"ID":"f3b0d918-b8bc-4367-9c8f-fe90da2e92be","Type":"ContainerStarted","Data":"362f6b3e4cc51b4cbd4878b73466bef0c6a86d665c09b3da4fd8ee42b1e6dbba"} Oct 30 13:08:56.826804 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:56.826682 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph" event={"ID":"f3b0d918-b8bc-4367-9c8f-fe90da2e92be","Type":"ContainerStarted","Data":"e149423d5818ba02adcbf42156f340b52d3789757bb8d743baed822f4100c9e0"} Oct 30 13:08:56.872446 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:56.872372 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph" podStartSLOduration=1.872359146 podStartE2EDuration="1.872359146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:56.869654959 +0000 UTC m=+4487.903793355" watchObservedRunningTime="2026-10-30 13:08:56.872359146 +0000 UTC m=+4487.906497541" Oct 30 13:08:57.332031 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.332000 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" Oct 30 13:08:57.352374 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.352353 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" Oct 30 13:08:57.479223 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.479196 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/179f5f3b-6ed4-4330-91ee-6dcbf00ea87a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/179f5f3b-6ed4-4330-91ee-6dcbf00ea87a-plugins-config-volume\") pod \"179f5f3b-6ed4-4330-91ee-6dcbf00ea87a\" (UID: \"179f5f3b-6ed4-4330-91ee-6dcbf00ea87a\") " Oct 30 13:08:57.479413 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.479245 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0022531e-0b86-46d8-b59e-ed23010efa07-kube-api-access-7mw5n\" (UniqueName: \"kubernetes.io/projected/0022531e-0b86-46d8-b59e-ed23010efa07-kube-api-access-7mw5n\") pod \"0022531e-0b86-46d8-b59e-ed23010efa07\" (UID: \"0022531e-0b86-46d8-b59e-ed23010efa07\") " Oct 30 13:08:57.479413 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.479368 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/179f5f3b-6ed4-4330-91ee-6dcbf00ea87a-kube-api-access-zr4sr\" (UniqueName: \"kubernetes.io/projected/179f5f3b-6ed4-4330-91ee-6dcbf00ea87a-kube-api-access-zr4sr\") pod \"179f5f3b-6ed4-4330-91ee-6dcbf00ea87a\" (UID: \"179f5f3b-6ed4-4330-91ee-6dcbf00ea87a\") " Oct 30 13:08:57.479413 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.479411 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0022531e-0b86-46d8-b59e-ed23010efa07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0022531e-0b86-46d8-b59e-ed23010efa07-plugins-config-volume\") pod \"0022531e-0b86-46d8-b59e-ed23010efa07\" (UID: \"0022531e-0b86-46d8-b59e-ed23010efa07\") " Oct 30 13:08:57.479601 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.479569 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/179f5f3b-6ed4-4330-91ee-6dcbf00ea87a-plugins-config-volume" pod "179f5f3b-6ed4-4330-91ee-6dcbf00ea87a" (UID: "179f5f3b-6ed4-4330-91ee-6dcbf00ea87a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:57.479801 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.479664 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/179f5f3b-6ed4-4330-91ee-6dcbf00ea87a-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:57.479889 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.479826 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0022531e-0b86-46d8-b59e-ed23010efa07-plugins-config-volume" pod "0022531e-0b86-46d8-b59e-ed23010efa07" (UID: "0022531e-0b86-46d8-b59e-ed23010efa07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:57.481424 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.481401 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179f5f3b-6ed4-4330-91ee-6dcbf00ea87a-kube-api-access-zr4sr" pod "179f5f3b-6ed4-4330-91ee-6dcbf00ea87a" (UID: "179f5f3b-6ed4-4330-91ee-6dcbf00ea87a"). InnerVolumeSpecName "kube-api-access-zr4sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:57.481519 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.481431 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0022531e-0b86-46d8-b59e-ed23010efa07-kube-api-access-7mw5n" pod "0022531e-0b86-46d8-b59e-ed23010efa07" (UID: "0022531e-0b86-46d8-b59e-ed23010efa07"). InnerVolumeSpecName "kube-api-access-7mw5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:57.581247 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.581223 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr4sr\" (UniqueName: \"kubernetes.io/projected/179f5f3b-6ed4-4330-91ee-6dcbf00ea87a-kube-api-access-zr4sr\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:57.581247 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.581245 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0022531e-0b86-46d8-b59e-ed23010efa07-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:57.581382 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.581254 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mw5n\" (UniqueName: \"kubernetes.io/projected/0022531e-0b86-46d8-b59e-ed23010efa07-kube-api-access-7mw5n\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:57.743251 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.743186 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" podUID="fbe0584e-df4b-4393-9313-9f347cc38767" containerName="payload-processing" containerID="cri-o://c5e801734b99770487116591d475290cf177a00eb649137635f83bb1a7e59e9b" gracePeriod=25 Oct 30 13:08:57.802972 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.802941 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" podUID="ca8bf37f-b34e-42b2-810e-96d117d30e13" containerName="payload-pre-processing" containerID="cri-o://2f9ad94cab59ed1f933f7057334fad07db2582961b38701d2dff27b550930844" gracePeriod=25 Oct 30 13:08:57.832524 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.832497 4468 generic.go:356] "Generic (PLEG): container finished" podID="f3b0d918-b8bc-4367-9c8f-fe90da2e92be" containerID="362f6b3e4cc51b4cbd4878b73466bef0c6a86d665c09b3da4fd8ee42b1e6dbba" exitCode=0 Oct 30 13:08:57.832618 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.832582 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph" event={"ID":"f3b0d918-b8bc-4367-9c8f-fe90da2e92be","Type":"ContainerDied","Data":"362f6b3e4cc51b4cbd4878b73466bef0c6a86d665c09b3da4fd8ee42b1e6dbba"} Oct 30 13:08:57.833790 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.833768 4468 generic.go:356] "Generic (PLEG): container finished" podID="0022531e-0b86-46d8-b59e-ed23010efa07" containerID="69c35757960310240363fde0857945765d07e5bfa89603294c2f3ed3f100cc7a" exitCode=0 Oct 30 13:08:57.833890 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.833806 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" Oct 30 13:08:57.833890 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.833847 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" event={"ID":"0022531e-0b86-46d8-b59e-ed23010efa07","Type":"ContainerDied","Data":"69c35757960310240363fde0857945765d07e5bfa89603294c2f3ed3f100cc7a"} Oct 30 13:08:57.834012 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.833898 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5" event={"ID":"0022531e-0b86-46d8-b59e-ed23010efa07","Type":"ContainerDied","Data":"073f338c05f8a7fb0a290aa9a245945668f8a6936d2eff6934fe717032dcfa71"} Oct 30 13:08:57.834012 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.833929 4468 scope.go:122] "RemoveContainer" containerID="69c35757960310240363fde0857945765d07e5bfa89603294c2f3ed3f100cc7a" Oct 30 13:08:57.835278 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.835254 4468 generic.go:356] "Generic (PLEG): container finished" podID="179f5f3b-6ed4-4330-91ee-6dcbf00ea87a" containerID="8f5cde7c40b46ed60cf63473b3f046c56748a71abbd67c87cff7fea392e06d5b" exitCode=0 Oct 30 13:08:57.835368 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.835287 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" Oct 30 13:08:57.835368 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.835324 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" event={"ID":"179f5f3b-6ed4-4330-91ee-6dcbf00ea87a","Type":"ContainerDied","Data":"8f5cde7c40b46ed60cf63473b3f046c56748a71abbd67c87cff7fea392e06d5b"} Oct 30 13:08:57.835368 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.835363 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz" event={"ID":"179f5f3b-6ed4-4330-91ee-6dcbf00ea87a","Type":"ContainerDied","Data":"7f981b084e3a717d87207093f0718dc82c0a4aa6800587ecbbf9c74e12d74e37"} Oct 30 13:08:57.843751 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.843716 4468 scope.go:122] "RemoveContainer" containerID="69c35757960310240363fde0857945765d07e5bfa89603294c2f3ed3f100cc7a" Oct 30 13:08:57.844047 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:08:57.844025 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c35757960310240363fde0857945765d07e5bfa89603294c2f3ed3f100cc7a\": container with ID starting with 69c35757960310240363fde0857945765d07e5bfa89603294c2f3ed3f100cc7a not found: ID does not exist" containerID="69c35757960310240363fde0857945765d07e5bfa89603294c2f3ed3f100cc7a" Oct 30 13:08:57.844122 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.844055 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c35757960310240363fde0857945765d07e5bfa89603294c2f3ed3f100cc7a"} err="failed to get container status \"69c35757960310240363fde0857945765d07e5bfa89603294c2f3ed3f100cc7a\": rpc error: code = NotFound desc = could not find container \"69c35757960310240363fde0857945765d07e5bfa89603294c2f3ed3f100cc7a\": container with ID starting with 69c35757960310240363fde0857945765d07e5bfa89603294c2f3ed3f100cc7a not found: ID does not exist" Oct 30 13:08:57.844122 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.844091 4468 scope.go:122] "RemoveContainer" containerID="8f5cde7c40b46ed60cf63473b3f046c56748a71abbd67c87cff7fea392e06d5b" Oct 30 13:08:57.852127 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.852111 4468 scope.go:122] "RemoveContainer" containerID="8f5cde7c40b46ed60cf63473b3f046c56748a71abbd67c87cff7fea392e06d5b" Oct 30 13:08:57.852379 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:08:57.852359 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5cde7c40b46ed60cf63473b3f046c56748a71abbd67c87cff7fea392e06d5b\": container with ID starting with 8f5cde7c40b46ed60cf63473b3f046c56748a71abbd67c87cff7fea392e06d5b not found: ID does not exist" containerID="8f5cde7c40b46ed60cf63473b3f046c56748a71abbd67c87cff7fea392e06d5b" Oct 30 13:08:57.852447 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.852388 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5cde7c40b46ed60cf63473b3f046c56748a71abbd67c87cff7fea392e06d5b"} err="failed to get container status \"8f5cde7c40b46ed60cf63473b3f046c56748a71abbd67c87cff7fea392e06d5b\": rpc error: code = NotFound desc = could not find container \"8f5cde7c40b46ed60cf63473b3f046c56748a71abbd67c87cff7fea392e06d5b\": container with ID starting with 8f5cde7c40b46ed60cf63473b3f046c56748a71abbd67c87cff7fea392e06d5b not found: ID does not exist" Oct 30 13:08:57.922681 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.922655 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5"] Oct 30 13:08:57.929099 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.927222 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-8d2170-64ff67b7f8-xcgn5"] Oct 30 13:08:57.963048 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.963020 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz"] Oct 30 13:08:57.968185 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:57.968162 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-8d2170-54f4bf7fcf-l2xkz"] Oct 30 13:08:58.985854 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:58.985823 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph" Oct 30 13:08:59.088495 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:59.088455 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0022531e-0b86-46d8-b59e-ed23010efa07" path="/var/lib/kubelet/pods/0022531e-0b86-46d8-b59e-ed23010efa07/volumes" Oct 30 13:08:59.088972 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:59.088950 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="179f5f3b-6ed4-4330-91ee-6dcbf00ea87a" path="/var/lib/kubelet/pods/179f5f3b-6ed4-4330-91ee-6dcbf00ea87a/volumes" Oct 30 13:08:59.096260 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:59.096203 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-maas-api-openshift-ca\") pod \"f3b0d918-b8bc-4367-9c8f-fe90da2e92be\" (UID: \"f3b0d918-b8bc-4367-9c8f-fe90da2e92be\") " Oct 30 13:08:59.096393 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:59.096294 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-kube-api-access-dmrdp\" (UniqueName: \"kubernetes.io/projected/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-kube-api-access-dmrdp\") pod \"f3b0d918-b8bc-4367-9c8f-fe90da2e92be\" (UID: \"f3b0d918-b8bc-4367-9c8f-fe90da2e92be\") " Oct 30 13:08:59.096393 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:59.096363 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-maas-api-xks-ca\") pod \"f3b0d918-b8bc-4367-9c8f-fe90da2e92be\" (UID: \"f3b0d918-b8bc-4367-9c8f-fe90da2e92be\") " Oct 30 13:08:59.096599 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:59.096576 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-maas-api-openshift-ca" pod "f3b0d918-b8bc-4367-9c8f-fe90da2e92be" (UID: "f3b0d918-b8bc-4367-9c8f-fe90da2e92be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:59.096735 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:59.096719 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-maas-api-openshift-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:59.098442 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:59.098409 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-maas-api-xks-ca" pod "f3b0d918-b8bc-4367-9c8f-fe90da2e92be" (UID: "f3b0d918-b8bc-4367-9c8f-fe90da2e92be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:59.098543 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:59.098468 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-kube-api-access-dmrdp" pod "f3b0d918-b8bc-4367-9c8f-fe90da2e92be" (UID: "f3b0d918-b8bc-4367-9c8f-fe90da2e92be"). InnerVolumeSpecName "kube-api-access-dmrdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:59.197514 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:59.197477 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmrdp\" (UniqueName: \"kubernetes.io/projected/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-kube-api-access-dmrdp\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:59.197514 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:59.197508 4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b0d918-b8bc-4367-9c8f-fe90da2e92be-maas-api-xks-ca\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:08:59.861093 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:59.857220 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph" event={"ID":"f3b0d918-b8bc-4367-9c8f-fe90da2e92be","Type":"ContainerDied","Data":"e149423d5818ba02adcbf42156f340b52d3789757bb8d743baed822f4100c9e0"} Oct 30 13:08:59.861093 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:59.857290 4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e149423d5818ba02adcbf42156f340b52d3789757bb8d743baed822f4100c9e0" Oct 30 13:08:59.861093 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:08:59.857364 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de18bc3-57lph" Oct 30 13:09:01.838400 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:01.838359 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" containerID="cri-o://ce78c872f3b60f0861af939896132f50f3dde93346456fa2c2943fa2681d8535" gracePeriod=45 Oct 30 13:09:01.843295 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:01.843260 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": EOF" Oct 30 13:09:01.848454 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:01.848423 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" containerID="cri-o://147fc8b34348ec09b51ed85b02b2b901d76a5e693f9aee7de87c8384c41a916e" gracePeriod=45 Oct 30 13:09:01.974217 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:01.974159 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": dial tcp 10.134.0.74:8000: connect: connection refused" Oct 30 13:09:02.092652 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.092591 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 13:09:02.118598 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.118573 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 13:09:02.123836 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.123813 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-home\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-home\") pod \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " Oct 30 13:09:02.123960 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.123866 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147053ac-cddc-4e20-ba45-ef81e5d3f719-kube-api-access-cvdnj\" (UniqueName: \"kubernetes.io/projected/147053ac-cddc-4e20-ba45-ef81e5d3f719-kube-api-access-cvdnj\") pod \"147053ac-cddc-4e20-ba45-ef81e5d3f719\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " Oct 30 13:09:02.123960 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.123906 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/147053ac-cddc-4e20-ba45-ef81e5d3f719-tls-certs\" (UniqueName: \"kubernetes.io/secret/147053ac-cddc-4e20-ba45-ef81e5d3f719-tls-certs\") pod \"147053ac-cddc-4e20-ba45-ef81e5d3f719\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " Oct 30 13:09:02.123960 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.123936 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea5e911-0baf-41f3-8ccc-39aa997f4151-kube-api-access-zjr4f\" (UniqueName: \"kubernetes.io/projected/2ea5e911-0baf-41f3-8ccc-39aa997f4151-kube-api-access-zjr4f\") pod \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " Oct 30 13:09:02.124155 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.123965 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-model-cache\") pod \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " Oct 30 13:09:02.124155 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.124097 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-home" pod "2ea5e911-0baf-41f3-8ccc-39aa997f4151" (UID: "2ea5e911-0baf-41f3-8ccc-39aa997f4151"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:02.124155 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.124099 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ea5e911-0baf-41f3-8ccc-39aa997f4151-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea5e911-0baf-41f3-8ccc-39aa997f4151-tls-certs\") pod \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " Oct 30 13:09:02.124325 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.124201 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-model-cache\") pod \"147053ac-cddc-4e20-ba45-ef81e5d3f719\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " Oct 30 13:09:02.124325 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.124253 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-model-cache" pod "2ea5e911-0baf-41f3-8ccc-39aa997f4151" (UID: "2ea5e911-0baf-41f3-8ccc-39aa997f4151"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:02.124325 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.124259 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-home\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-home\") pod \"147053ac-cddc-4e20-ba45-ef81e5d3f719\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " Oct 30 13:09:02.124483 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.124359 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-dshm\") pod \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " Oct 30 13:09:02.124483 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.124442 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-home" pod "147053ac-cddc-4e20-ba45-ef81e5d3f719" (UID: "147053ac-cddc-4e20-ba45-ef81e5d3f719"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:02.124603 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.124449 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-tmp-dir\") pod \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\" (UID: \"2ea5e911-0baf-41f3-8ccc-39aa997f4151\") " Oct 30 13:09:02.124603 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.124513 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-model-cache" pod "147053ac-cddc-4e20-ba45-ef81e5d3f719" (UID: "147053ac-cddc-4e20-ba45-ef81e5d3f719"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:02.124603 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.124539 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-dshm\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-dshm\") pod \"147053ac-cddc-4e20-ba45-ef81e5d3f719\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " Oct 30 13:09:02.124603 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.124582 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-tmp-dir\") pod \"147053ac-cddc-4e20-ba45-ef81e5d3f719\" (UID: \"147053ac-cddc-4e20-ba45-ef81e5d3f719\") " Oct 30 13:09:02.124803 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.124633 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-tmp-dir" pod "2ea5e911-0baf-41f3-8ccc-39aa997f4151" (UID: "2ea5e911-0baf-41f3-8ccc-39aa997f4151"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:02.124910 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.124892 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-tmp-dir" pod "147053ac-cddc-4e20-ba45-ef81e5d3f719" (UID: "147053ac-cddc-4e20-ba45-ef81e5d3f719"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:02.125002 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.124982 4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-tmp-dir\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.125083 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.125008 4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-home\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.125083 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.125021 4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-model-cache\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.125083 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.125033 4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-model-cache\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.125083 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.125046 4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-home\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.125083 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.125058 4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-tmp-dir\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.126476 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.126448 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147053ac-cddc-4e20-ba45-ef81e5d3f719-kube-api-access-cvdnj" pod "147053ac-cddc-4e20-ba45-ef81e5d3f719" (UID: "147053ac-cddc-4e20-ba45-ef81e5d3f719"). InnerVolumeSpecName "kube-api-access-cvdnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:02.126613 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.126468 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/147053ac-cddc-4e20-ba45-ef81e5d3f719-tls-certs" pod "147053ac-cddc-4e20-ba45-ef81e5d3f719" (UID: "147053ac-cddc-4e20-ba45-ef81e5d3f719"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:09:02.126710 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.126692 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-dshm" pod "2ea5e911-0baf-41f3-8ccc-39aa997f4151" (UID: "2ea5e911-0baf-41f3-8ccc-39aa997f4151"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:02.126804 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.126790 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ea5e911-0baf-41f3-8ccc-39aa997f4151-tls-certs" pod "2ea5e911-0baf-41f3-8ccc-39aa997f4151" (UID: "2ea5e911-0baf-41f3-8ccc-39aa997f4151"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:09:02.126878 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.126822 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea5e911-0baf-41f3-8ccc-39aa997f4151-kube-api-access-zjr4f" pod "2ea5e911-0baf-41f3-8ccc-39aa997f4151" (UID: "2ea5e911-0baf-41f3-8ccc-39aa997f4151"). InnerVolumeSpecName "kube-api-access-zjr4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:02.127159 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.127137 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-dshm" pod "147053ac-cddc-4e20-ba45-ef81e5d3f719" (UID: "147053ac-cddc-4e20-ba45-ef81e5d3f719"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:02.226110 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.226042 4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/147053ac-cddc-4e20-ba45-ef81e5d3f719-dshm\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.226279 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.226119 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvdnj\" (UniqueName: \"kubernetes.io/projected/147053ac-cddc-4e20-ba45-ef81e5d3f719-kube-api-access-cvdnj\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.226279 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.226137 4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147053ac-cddc-4e20-ba45-ef81e5d3f719-tls-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.226279 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.226150 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjr4f\" (UniqueName: \"kubernetes.io/projected/2ea5e911-0baf-41f3-8ccc-39aa997f4151-kube-api-access-zjr4f\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.226279 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.226163 4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea5e911-0baf-41f3-8ccc-39aa997f4151-tls-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.226279 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.226175 4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea5e911-0baf-41f3-8ccc-39aa997f4151-dshm\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.871611 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.871574 4468 generic.go:356] "Generic (PLEG): container finished" podID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerID="ce78c872f3b60f0861af939896132f50f3dde93346456fa2c2943fa2681d8535" exitCode=0 Oct 30 13:09:02.872027 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.871630 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" Oct 30 13:09:02.872027 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.871652 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" event={"ID":"147053ac-cddc-4e20-ba45-ef81e5d3f719","Type":"ContainerDied","Data":"ce78c872f3b60f0861af939896132f50f3dde93346456fa2c2943fa2681d8535"} Oct 30 13:09:02.872027 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.871694 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5" event={"ID":"147053ac-cddc-4e20-ba45-ef81e5d3f719","Type":"ContainerDied","Data":"a3f09e1b896586cd3ba0cff57c23f02ea6d0657e5212759660204306c77ee3c0"} Oct 30 13:09:02.872027 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.871717 4468 scope.go:122] "RemoveContainer" containerID="ce78c872f3b60f0861af939896132f50f3dde93346456fa2c2943fa2681d8535" Oct 30 13:09:02.873410 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.873380 4468 generic.go:356] "Generic (PLEG): container finished" podID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerID="147fc8b34348ec09b51ed85b02b2b901d76a5e693f9aee7de87c8384c41a916e" exitCode=0 Oct 30 13:09:02.873486 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.873419 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" Oct 30 13:09:02.873486 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.873451 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" event={"ID":"2ea5e911-0baf-41f3-8ccc-39aa997f4151","Type":"ContainerDied","Data":"147fc8b34348ec09b51ed85b02b2b901d76a5e693f9aee7de87c8384c41a916e"} Oct 30 13:09:02.873486 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.873480 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5" event={"ID":"2ea5e911-0baf-41f3-8ccc-39aa997f4151","Type":"ContainerDied","Data":"c4da4b45429675dd774aad3bebcd679cf9acfbe73259d87e33cbdb1e8fb8e4d6"} Oct 30 13:09:02.881919 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.881764 4468 scope.go:122] "RemoveContainer" containerID="ce78c872f3b60f0861af939896132f50f3dde93346456fa2c2943fa2681d8535" Oct 30 13:09:02.882181 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:09:02.882144 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce78c872f3b60f0861af939896132f50f3dde93346456fa2c2943fa2681d8535\": container with ID starting with ce78c872f3b60f0861af939896132f50f3dde93346456fa2c2943fa2681d8535 not found: ID does not exist" containerID="ce78c872f3b60f0861af939896132f50f3dde93346456fa2c2943fa2681d8535" Oct 30 13:09:02.882302 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.882184 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce78c872f3b60f0861af939896132f50f3dde93346456fa2c2943fa2681d8535"} err="failed to get container status \"ce78c872f3b60f0861af939896132f50f3dde93346456fa2c2943fa2681d8535\": rpc error: code = NotFound desc = could not find container \"ce78c872f3b60f0861af939896132f50f3dde93346456fa2c2943fa2681d8535\": container with ID starting with ce78c872f3b60f0861af939896132f50f3dde93346456fa2c2943fa2681d8535 not found: ID does not exist" Oct 30 13:09:02.882302 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.882201 4468 scope.go:122] "RemoveContainer" containerID="147fc8b34348ec09b51ed85b02b2b901d76a5e693f9aee7de87c8384c41a916e" Oct 30 13:09:02.890114 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.890098 4468 scope.go:122] "RemoveContainer" containerID="147fc8b34348ec09b51ed85b02b2b901d76a5e693f9aee7de87c8384c41a916e" Oct 30 13:09:02.890365 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:09:02.890347 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"147fc8b34348ec09b51ed85b02b2b901d76a5e693f9aee7de87c8384c41a916e\": container with ID starting with 147fc8b34348ec09b51ed85b02b2b901d76a5e693f9aee7de87c8384c41a916e not found: ID does not exist" containerID="147fc8b34348ec09b51ed85b02b2b901d76a5e693f9aee7de87c8384c41a916e" Oct 30 13:09:02.890403 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.890372 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"147fc8b34348ec09b51ed85b02b2b901d76a5e693f9aee7de87c8384c41a916e"} err="failed to get container status \"147fc8b34348ec09b51ed85b02b2b901d76a5e693f9aee7de87c8384c41a916e\": rpc error: code = NotFound desc = could not find container \"147fc8b34348ec09b51ed85b02b2b901d76a5e693f9aee7de87c8384c41a916e\": container with ID starting with 147fc8b34348ec09b51ed85b02b2b901d76a5e693f9aee7de87c8384c41a916e not found: ID does not exist" Oct 30 13:09:02.929791 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.929771 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5"] Oct 30 13:09:02.943299 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.943277 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/premium-w4-512ec5-kserve-b784cb9cf-2xxr5"] Oct 30 13:09:02.980699 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.980677 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5"] Oct 30 13:09:02.984973 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:02.984955 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/distinct-w4-512ec5-kserve-7b7dddcb4b-z52v5"] Oct 30 13:09:03.087622 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:03.087598 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" path="/var/lib/kubelet/pods/147053ac-cddc-4e20-ba45-ef81e5d3f719/volumes" Oct 30 13:09:03.087996 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:03.087983 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" path="/var/lib/kubelet/pods/2ea5e911-0baf-41f3-8ccc-39aa997f4151/volumes" Oct 30 13:09:03.555343 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:03.555308 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" podUID="fbe0584e-df4b-4393-9313-9f347cc38767" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:09:04.567198 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:04.567163 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" podUID="ca8bf37f-b34e-42b2-810e-96d117d30e13" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:09:11.549768 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:11.549728 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn"] Oct 30 13:09:11.562461 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:11.562434 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-512ec5-tf9vn"] Oct 30 13:09:13.088482 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:13.088452 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8" path="/var/lib/kubelet/pods/a3a3bc74-bfc3-4ccb-ab1b-99dc6f3c0db8/volumes" Oct 30 13:09:13.555974 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:13.555939 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" podUID="fbe0584e-df4b-4393-9313-9f347cc38767" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:09:14.566842 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:14.566802 4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" podUID="ca8bf37f-b34e-42b2-810e-96d117d30e13" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:09:17.945525 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:17.945481 4468 generic.go:356] "Generic (PLEG): container finished" podID="ca8bf37f-b34e-42b2-810e-96d117d30e13" containerID="2f9ad94cab59ed1f933f7057334fad07db2582961b38701d2dff27b550930844" exitCode=0 Oct 30 13:09:17.945977 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:17.945551 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" event={"ID":"ca8bf37f-b34e-42b2-810e-96d117d30e13","Type":"ContainerDied","Data":"2f9ad94cab59ed1f933f7057334fad07db2582961b38701d2dff27b550930844"} Oct 30 13:09:17.947363 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:17.947337 4468 generic.go:356] "Generic (PLEG): container finished" podID="fbe0584e-df4b-4393-9313-9f347cc38767" containerID="c5e801734b99770487116591d475290cf177a00eb649137635f83bb1a7e59e9b" exitCode=0 Oct 30 13:09:17.947470 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:17.947414 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" event={"ID":"fbe0584e-df4b-4393-9313-9f347cc38767","Type":"ContainerDied","Data":"c5e801734b99770487116591d475290cf177a00eb649137635f83bb1a7e59e9b"} Oct 30 13:09:18.050897 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.050868 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" Oct 30 13:09:18.076079 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.076047 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" Oct 30 13:09:18.164688 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.164662 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca8bf37f-b34e-42b2-810e-96d117d30e13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8bf37f-b34e-42b2-810e-96d117d30e13-plugins-config-volume\") pod \"ca8bf37f-b34e-42b2-810e-96d117d30e13\" (UID: \"ca8bf37f-b34e-42b2-810e-96d117d30e13\") " Oct 30 13:09:18.164848 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.164745 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca8bf37f-b34e-42b2-810e-96d117d30e13-kube-api-access-ltgvk\" (UniqueName: \"kubernetes.io/projected/ca8bf37f-b34e-42b2-810e-96d117d30e13-kube-api-access-ltgvk\") pod \"ca8bf37f-b34e-42b2-810e-96d117d30e13\" (UID: \"ca8bf37f-b34e-42b2-810e-96d117d30e13\") " Oct 30 13:09:18.164848 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.164780 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbe0584e-df4b-4393-9313-9f347cc38767-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe0584e-df4b-4393-9313-9f347cc38767-plugins-config-volume\") pod \"fbe0584e-df4b-4393-9313-9f347cc38767\" (UID: \"fbe0584e-df4b-4393-9313-9f347cc38767\") " Oct 30 13:09:18.164985 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.164865 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe0584e-df4b-4393-9313-9f347cc38767-kube-api-access-z9t2s\" (UniqueName: \"kubernetes.io/projected/fbe0584e-df4b-4393-9313-9f347cc38767-kube-api-access-z9t2s\") pod \"fbe0584e-df4b-4393-9313-9f347cc38767\" (UID: \"fbe0584e-df4b-4393-9313-9f347cc38767\") " Oct 30 13:09:18.165141 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.165116 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca8bf37f-b34e-42b2-810e-96d117d30e13-plugins-config-volume" pod "ca8bf37f-b34e-42b2-810e-96d117d30e13" (UID: "ca8bf37f-b34e-42b2-810e-96d117d30e13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:09:18.165239 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.165224 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8bf37f-b34e-42b2-810e-96d117d30e13-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:18.165286 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.165254 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe0584e-df4b-4393-9313-9f347cc38767-plugins-config-volume" pod "fbe0584e-df4b-4393-9313-9f347cc38767" (UID: "fbe0584e-df4b-4393-9313-9f347cc38767"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:09:18.166902 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.166878 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8bf37f-b34e-42b2-810e-96d117d30e13-kube-api-access-ltgvk" pod "ca8bf37f-b34e-42b2-810e-96d117d30e13" (UID: "ca8bf37f-b34e-42b2-810e-96d117d30e13"). InnerVolumeSpecName "kube-api-access-ltgvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:18.166994 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.166921 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe0584e-df4b-4393-9313-9f347cc38767-kube-api-access-z9t2s" pod "fbe0584e-df4b-4393-9313-9f347cc38767" (UID: "fbe0584e-df4b-4393-9313-9f347cc38767"). InnerVolumeSpecName "kube-api-access-z9t2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:18.266105 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.266039 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9t2s\" (UniqueName: \"kubernetes.io/projected/fbe0584e-df4b-4393-9313-9f347cc38767-kube-api-access-z9t2s\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:18.266105 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.266060 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltgvk\" (UniqueName: \"kubernetes.io/projected/ca8bf37f-b34e-42b2-810e-96d117d30e13-kube-api-access-ltgvk\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:18.266105 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.266082 4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe0584e-df4b-4393-9313-9f347cc38767-plugins-config-volume\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:18.874874 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.874832 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb"] Oct 30 13:09:18.875108 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.875078 4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" podUID="e1a20ae7-502b-447e-ae58-120744dd79fa" containerName="istio-proxy" containerID="cri-o://5ab9aa8b29a34505d45a1d36317486f74ab18388e30ec25c8863d37e4a5dcfd8" gracePeriod=30 Oct 30 13:09:18.956108 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.953239 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" Oct 30 13:09:18.956108 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.955205 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2" event={"ID":"ca8bf37f-b34e-42b2-810e-96d117d30e13","Type":"ContainerDied","Data":"a427360fbb40a08453346472696a751bbd807756f29c2e379d912ab5b24604d3"} Oct 30 13:09:18.956108 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.955262 4468 scope.go:122] "RemoveContainer" containerID="2f9ad94cab59ed1f933f7057334fad07db2582961b38701d2dff27b550930844" Oct 30 13:09:18.958534 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.958506 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" event={"ID":"fbe0584e-df4b-4393-9313-9f347cc38767","Type":"ContainerDied","Data":"80fa79dfd7a767a1dc38729319c44e7427f59bf6c3fd2319dfddbf45396e63db"} Oct 30 13:09:18.958713 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.958537 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt" Oct 30 13:09:18.967330 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:18.967313 4468 scope.go:122] "RemoveContainer" containerID="c5e801734b99770487116591d475290cf177a00eb649137635f83bb1a7e59e9b" Oct 30 13:09:19.016838 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:19.016761 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2"] Oct 30 13:09:19.020614 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:19.020583 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-512ec5-5d778778f9-d4dw2"] Oct 30 13:09:19.069959 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:19.069938 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt"] Oct 30 13:09:19.075108 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:19.075087 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-512ec5-5dd4cfdc96-4c8qt"] Oct 30 13:09:19.087810 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:19.087788 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca8bf37f-b34e-42b2-810e-96d117d30e13" path="/var/lib/kubelet/pods/ca8bf37f-b34e-42b2-810e-96d117d30e13/volumes" Oct 30 13:09:19.088179 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:19.088165 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe0584e-df4b-4393-9313-9f347cc38767" path="/var/lib/kubelet/pods/fbe0584e-df4b-4393-9313-9f347cc38767/volumes" Oct 30 13:09:20.043436 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:20.043410 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5949f4fc4c-bpf6c_0d4dab63-54cf-49ea-98a6-cfbf3adc9b68/authorino/0.log" Oct 30 13:09:22.459982 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:22.459946 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6858bfbcf8-h9qws_bc2e5679-e297-4aae-8676-b55f902809b1/manager/0.log" Oct 30 13:09:23.607453 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:09:23.607409 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:09:23.607970 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:23.607950 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb8f9449b-nnd2b_731fb32a-31aa-4d57-8edd-4f890b9e9288/manager/1.log" Oct 30 13:09:24.126199 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.126172 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 13:09:24.217215 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.217189 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-token\" (UniqueName: \"kubernetes.io/projected/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-token\") pod \"e1a20ae7-502b-447e-ae58-120744dd79fa\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " Oct 30 13:09:24.217383 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.217259 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-workload-socket\") pod \"e1a20ae7-502b-447e-ae58-120744dd79fa\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " Oct 30 13:09:24.217383 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.217279 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-envoy\") pod \"e1a20ae7-502b-447e-ae58-120744dd79fa\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " Oct 30 13:09:24.217383 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.217311 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1a20ae7-502b-447e-ae58-120744dd79fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1a20ae7-502b-447e-ae58-120744dd79fa-istiod-ca-cert\") pod \"e1a20ae7-502b-447e-ae58-120744dd79fa\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " Oct 30 13:09:24.217383 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.217333 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-credential-socket\") pod \"e1a20ae7-502b-447e-ae58-120744dd79fa\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " Oct 30 13:09:24.217383 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.217378 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a20ae7-502b-447e-ae58-120744dd79fa-kube-api-access-fxzmr\" (UniqueName: \"kubernetes.io/projected/e1a20ae7-502b-447e-ae58-120744dd79fa-kube-api-access-fxzmr\") pod \"e1a20ae7-502b-447e-ae58-120744dd79fa\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " Oct 30 13:09:24.217677 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.217403 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-data\") pod \"e1a20ae7-502b-447e-ae58-120744dd79fa\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " Oct 30 13:09:24.217677 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.217431 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-workload-certs\") pod \"e1a20ae7-502b-447e-ae58-120744dd79fa\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " Oct 30 13:09:24.217677 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.217465 4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-podinfo\") pod \"e1a20ae7-502b-447e-ae58-120744dd79fa\" (UID: \"e1a20ae7-502b-447e-ae58-120744dd79fa\") " Oct 30 13:09:24.217677 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.217650 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-workload-socket" pod "e1a20ae7-502b-447e-ae58-120744dd79fa" (UID: "e1a20ae7-502b-447e-ae58-120744dd79fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:24.217894 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.217669 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-credential-socket" pod "e1a20ae7-502b-447e-ae58-120744dd79fa" (UID: "e1a20ae7-502b-447e-ae58-120744dd79fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:24.217894 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.217706 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a20ae7-502b-447e-ae58-120744dd79fa-istiod-ca-cert" pod "e1a20ae7-502b-447e-ae58-120744dd79fa" (UID: "e1a20ae7-502b-447e-ae58-120744dd79fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:09:24.218018 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.217964 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-workload-certs" pod "e1a20ae7-502b-447e-ae58-120744dd79fa" (UID: "e1a20ae7-502b-447e-ae58-120744dd79fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:24.218111 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.218049 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-data" pod "e1a20ae7-502b-447e-ae58-120744dd79fa" (UID: "e1a20ae7-502b-447e-ae58-120744dd79fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:24.218256 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.218235 4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-data\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:24.218326 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.218262 4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-workload-certs\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:24.218326 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.218276 4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-workload-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:24.218326 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.218289 4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1a20ae7-502b-447e-ae58-120744dd79fa-istiod-ca-cert\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:24.218326 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.218303 4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-credential-socket\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:24.219809 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.219780 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-podinfo" pod "e1a20ae7-502b-447e-ae58-120744dd79fa" (UID: "e1a20ae7-502b-447e-ae58-120744dd79fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:09:24.219931 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.219849 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-envoy" pod "e1a20ae7-502b-447e-ae58-120744dd79fa" (UID: "e1a20ae7-502b-447e-ae58-120744dd79fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:24.219931 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.219869 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-token" pod "e1a20ae7-502b-447e-ae58-120744dd79fa" (UID: "e1a20ae7-502b-447e-ae58-120744dd79fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:24.220169 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.220148 4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a20ae7-502b-447e-ae58-120744dd79fa-kube-api-access-fxzmr" pod "e1a20ae7-502b-447e-ae58-120744dd79fa" (UID: "e1a20ae7-502b-447e-ae58-120744dd79fa"). InnerVolumeSpecName "kube-api-access-fxzmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:24.243512 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.243490 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-j7nfs_fb5b969c-76f1-47a1-8fc0-00fac17f17d0/proxy/0.log" Oct 30 13:09:24.318998 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.318976 4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-podinfo\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:24.318998 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.318997 4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-token\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:24.319160 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.319006 4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1a20ae7-502b-447e-ae58-120744dd79fa-istio-envoy\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:24.319160 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.319013 4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxzmr\" (UniqueName: \"kubernetes.io/projected/e1a20ae7-502b-447e-ae58-120744dd79fa-kube-api-access-fxzmr\") on node \"ip-10-0-131-76.ec2.internal\" DevicePath \"\"" Oct 30 13:09:24.729648 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.729618 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb8f9449b-nnd2b_731fb32a-31aa-4d57-8edd-4f890b9e9288/manager/0.log" Oct 30 13:09:24.986720 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.986650 4468 generic.go:356] "Generic (PLEG): container finished" podID="e1a20ae7-502b-447e-ae58-120744dd79fa" containerID="5ab9aa8b29a34505d45a1d36317486f74ab18388e30ec25c8863d37e4a5dcfd8" exitCode=0 Oct 30 13:09:24.986720 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.986699 4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" Oct 30 13:09:24.986901 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.986732 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" event={"ID":"e1a20ae7-502b-447e-ae58-120744dd79fa","Type":"ContainerDied","Data":"5ab9aa8b29a34505d45a1d36317486f74ab18388e30ec25c8863d37e4a5dcfd8"} Oct 30 13:09:24.986901 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.986785 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb" event={"ID":"e1a20ae7-502b-447e-ae58-120744dd79fa","Type":"ContainerDied","Data":"6c5b602b3b19ef35719bc39ad04bb26d4799ce2db202201cc0bdf34451aa9d21"} Oct 30 13:09:24.986901 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.986808 4468 scope.go:122] "RemoveContainer" containerID="5ab9aa8b29a34505d45a1d36317486f74ab18388e30ec25c8863d37e4a5dcfd8" Oct 30 13:09:24.996010 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.995813 4468 scope.go:122] "RemoveContainer" containerID="5ab9aa8b29a34505d45a1d36317486f74ab18388e30ec25c8863d37e4a5dcfd8" Oct 30 13:09:24.996137 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:09:24.996116 4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab9aa8b29a34505d45a1d36317486f74ab18388e30ec25c8863d37e4a5dcfd8\": container with ID starting with 5ab9aa8b29a34505d45a1d36317486f74ab18388e30ec25c8863d37e4a5dcfd8 not found: ID does not exist" containerID="5ab9aa8b29a34505d45a1d36317486f74ab18388e30ec25c8863d37e4a5dcfd8" Oct 30 13:09:24.996191 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:24.996146 4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab9aa8b29a34505d45a1d36317486f74ab18388e30ec25c8863d37e4a5dcfd8"} err="failed to get container status \"5ab9aa8b29a34505d45a1d36317486f74ab18388e30ec25c8863d37e4a5dcfd8\": rpc error: code = NotFound desc = could not find container \"5ab9aa8b29a34505d45a1d36317486f74ab18388e30ec25c8863d37e4a5dcfd8\": container with ID starting with 5ab9aa8b29a34505d45a1d36317486f74ab18388e30ec25c8863d37e4a5dcfd8 not found: ID does not exist" Oct 30 13:09:25.059372 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:25.059348 4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb"] Oct 30 13:09:25.072666 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:25.072635 4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-dfgzb"] Oct 30 13:09:25.087598 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:25.087568 4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1a20ae7-502b-447e-ae58-120744dd79fa" path="/var/lib/kubelet/pods/e1a20ae7-502b-447e-ae58-120744dd79fa/volumes" Oct 30 13:09:25.281722 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:25.281642 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5949f4fc4c-bpf6c_0d4dab63-54cf-49ea-98a6-cfbf3adc9b68/authorino/0.log" Oct 30 13:09:36.093294 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:36.093266 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6858bfbcf8-h9qws_bc2e5679-e297-4aae-8676-b55f902809b1/manager/0.log" Oct 30 13:09:37.192426 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:37.192399 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb8f9449b-nnd2b_731fb32a-31aa-4d57-8edd-4f890b9e9288/manager/1.log" Oct 30 13:09:37.895119 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:37.895089 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-j7nfs_fb5b969c-76f1-47a1-8fc0-00fac17f17d0/proxy/0.log" Oct 30 13:09:38.351830 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:38.351795 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb8f9449b-nnd2b_731fb32a-31aa-4d57-8edd-4f890b9e9288/manager/0.log" Oct 30 13:09:39.336352 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:39.336324 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2de18bc3-57lph_f3b0d918-b8bc-4367-9c8f-fe90da2e92be/revoke-keys/0.log" Oct 30 13:09:39.903103 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:39.903060 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4e9110a7-tslmf_2bf6ae69-b68b-402b-bbd4-0846700c8962/revoke-keys/0.log" Oct 30 13:09:40.466913 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:40.466876 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6c99c3aa-rt7jk_80712f3d-cc77-4955-86ce-3c66d7aac999/revoke-keys/0.log" Oct 30 13:09:41.480643 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:41.480612 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-48ksl_6dd92beb-9078-4354-aa07-01d6ccf44643/postgres/0.log" Oct 30 13:09:43.009892 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:43.009844 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5949f4fc4c-bpf6c_0d4dab63-54cf-49ea-98a6-cfbf3adc9b68/authorino/0.log" Oct 30 13:09:45.715293 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:45.715249 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68fcc45f6c-fbxv7_b4d73c96-8ad8-4aba-908c-88f4969524c1/kube-auth-proxy/0.log" Oct 30 13:09:46.100731 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:46.100630 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-7tbjq_d6305f49-03f1-4d47-a0a7-19776c672543/istio-proxy/0.log" Oct 30 13:09:46.902522 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:46.902491 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-94zmh_a1d28f84-07c8-4c39-830b-608b124282f5/payload-pre-processing/0.log" Oct 30 13:09:47.098448 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:47.098403 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-ftmv4_910664f0-5127-40e1-b2c8-493f050457cc/payload-processing/0.log" Oct 30 13:09:48.054109 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:48.054080 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-st46v_ede14942-b51f-4e10-94cc-153445945351/main/0.log" Oct 30 13:09:48.259451 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:48.259400 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-5ltm6_75ebfa71-f8e5-4c79-bf68-1cee6c31e544/main/0.log" Oct 30 13:09:48.657615 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:48.657585 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-2gm67_6abf7dd7-2cd8-463a-95a6-1c5b3d391f73/main/0.log" Oct 30 13:09:49.422151 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:09:49.422098 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-b2d9z_f48b6092-fcf0-4787-adef-dba14c5df40f/main/0.log" Oct 30 13:09:53.597538 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:09:53.597459 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:10:00.258678 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.258644 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wls7l/must-gather-hfrq7"] Oct 30 13:10:00.259195 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259027 4468 cpu_manager.go:414] "containerMap: removing container" podUID="fbe0584e-df4b-4393-9313-9f347cc38767" containerName="payload-processing" Oct 30 13:10:00.259195 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259039 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe0584e-df4b-4393-9313-9f347cc38767" containerName="payload-processing" Oct 30 13:10:00.259195 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259056 4468 cpu_manager.go:414] "containerMap: removing container" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" Oct 30 13:10:00.259195 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259062 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" Oct 30 13:10:00.259195 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259086 4468 cpu_manager.go:414] "containerMap: removing container" podUID="ca8bf37f-b34e-42b2-810e-96d117d30e13" containerName="payload-pre-processing" Oct 30 13:10:00.259195 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259094 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca8bf37f-b34e-42b2-810e-96d117d30e13" containerName="payload-pre-processing" Oct 30 13:10:00.259195 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259105 4468 cpu_manager.go:414] "containerMap: removing container" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" Oct 30 13:10:00.259195 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259112 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" Oct 30 13:10:00.259195 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259123 4468 cpu_manager.go:414] "containerMap: removing container" podUID="e1a20ae7-502b-447e-ae58-120744dd79fa" containerName="istio-proxy" Oct 30 13:10:00.259195 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259129 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a20ae7-502b-447e-ae58-120744dd79fa" containerName="istio-proxy" Oct 30 13:10:00.259195 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259146 4468 cpu_manager.go:414] "containerMap: removing container" podUID="179f5f3b-6ed4-4330-91ee-6dcbf00ea87a" containerName="payload-pre-processing" Oct 30 13:10:00.259195 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259150 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="179f5f3b-6ed4-4330-91ee-6dcbf00ea87a" containerName="payload-pre-processing" Oct 30 13:10:00.259195 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259163 4468 cpu_manager.go:414] "containerMap: removing container" podUID="0022531e-0b86-46d8-b59e-ed23010efa07" containerName="payload-processing" Oct 30 13:10:00.259195 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259168 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0022531e-0b86-46d8-b59e-ed23010efa07" containerName="payload-processing" Oct 30 13:10:00.259195 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259182 4468 cpu_manager.go:414] "containerMap: removing container" podUID="f3b0d918-b8bc-4367-9c8f-fe90da2e92be" containerName="revoke-keys" Oct 30 13:10:00.259195 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259187 4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b0d918-b8bc-4367-9c8f-fe90da2e92be" containerName="revoke-keys" Oct 30 13:10:00.259866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259249 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="179f5f3b-6ed4-4330-91ee-6dcbf00ea87a" containerName="payload-pre-processing" Oct 30 13:10:00.259866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259259 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0022531e-0b86-46d8-b59e-ed23010efa07" containerName="payload-processing" Oct 30 13:10:00.259866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259267 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b0d918-b8bc-4367-9c8f-fe90da2e92be" containerName="revoke-keys" Oct 30 13:10:00.259866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259274 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe0584e-df4b-4393-9313-9f347cc38767" containerName="payload-processing" Oct 30 13:10:00.259866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259281 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="147053ac-cddc-4e20-ba45-ef81e5d3f719" containerName="main" Oct 30 13:10:00.259866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259287 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca8bf37f-b34e-42b2-810e-96d117d30e13" containerName="payload-pre-processing" Oct 30 13:10:00.259866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259292 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea5e911-0baf-41f3-8ccc-39aa997f4151" containerName="main" Oct 30 13:10:00.259866 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.259297 4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a20ae7-502b-447e-ae58-120744dd79fa" containerName="istio-proxy" Oct 30 13:10:00.262217 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.262195 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wls7l/must-gather-hfrq7" Oct 30 13:10:00.263958 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.263937 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t4cqb/must-gather-cl655"] Oct 30 13:10:00.267422 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.267404 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t4cqb/must-gather-cl655" Oct 30 13:10:00.282227 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.282207 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wls7l\"/\"default-dockercfg-psswm\"" Oct 30 13:10:00.295647 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.295624 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t4cqb\"/\"default-dockercfg-vplxv\"" Oct 30 13:10:00.295949 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.295927 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t4cqb\"/\"openshift-service-ca.crt\"" Oct 30 13:10:00.295949 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.295941 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wls7l\"/\"openshift-service-ca.crt\"" Oct 30 13:10:00.296148 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.295975 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wls7l\"/\"kube-root-ca.crt\"" Oct 30 13:10:00.296148 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.295975 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t4cqb\"/\"kube-root-ca.crt\"" Oct 30 13:10:00.298624 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.298604 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wls7l/must-gather-hfrq7"] Oct 30 13:10:00.300913 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.300893 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t4cqb/must-gather-cl655"] Oct 30 13:10:00.302474 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.302454 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjsrh\" (UniqueName: \"kubernetes.io/projected/86f0d9f7-b36b-4129-8360-a6ed3dfcf610-kube-api-access-zjsrh\") pod \"must-gather-hfrq7\" (UID: \"86f0d9f7-b36b-4129-8360-a6ed3dfcf610\") " pod="openshift-must-gather-wls7l/must-gather-hfrq7" Oct 30 13:10:00.302559 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.302489 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/958c13ff-fd33-4c6c-831d-483b6acb4ec5-must-gather-output\") pod \"must-gather-cl655\" (UID: \"958c13ff-fd33-4c6c-831d-483b6acb4ec5\") " pod="openshift-must-gather-t4cqb/must-gather-cl655" Oct 30 13:10:00.302559 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.302519 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82ql9\" (UniqueName: \"kubernetes.io/projected/958c13ff-fd33-4c6c-831d-483b6acb4ec5-kube-api-access-82ql9\") pod \"must-gather-cl655\" (UID: \"958c13ff-fd33-4c6c-831d-483b6acb4ec5\") " pod="openshift-must-gather-t4cqb/must-gather-cl655" Oct 30 13:10:00.302641 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.302616 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/86f0d9f7-b36b-4129-8360-a6ed3dfcf610-must-gather-output\") pod \"must-gather-hfrq7\" (UID: \"86f0d9f7-b36b-4129-8360-a6ed3dfcf610\") " pod="openshift-must-gather-wls7l/must-gather-hfrq7" Oct 30 13:10:00.353494 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.353472 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-47mpn/must-gather-vsvdr"] Oct 30 13:10:00.356924 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.356911 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47mpn/must-gather-vsvdr" Oct 30 13:10:00.375999 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.375975 4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-47mpn\"/\"default-dockercfg-xw6qs\"" Oct 30 13:10:00.376638 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.376616 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47mpn/must-gather-vsvdr"] Oct 30 13:10:00.382868 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.382850 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47mpn\"/\"openshift-service-ca.crt\"" Oct 30 13:10:00.383127 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.383113 4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47mpn\"/\"kube-root-ca.crt\"" Oct 30 13:10:00.403981 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.403955 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/958c13ff-fd33-4c6c-831d-483b6acb4ec5-must-gather-output\") pod \"must-gather-cl655\" (UID: \"958c13ff-fd33-4c6c-831d-483b6acb4ec5\") " pod="openshift-must-gather-t4cqb/must-gather-cl655" Oct 30 13:10:00.404081 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.403985 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82ql9\" (UniqueName: \"kubernetes.io/projected/958c13ff-fd33-4c6c-831d-483b6acb4ec5-kube-api-access-82ql9\") pod \"must-gather-cl655\" (UID: \"958c13ff-fd33-4c6c-831d-483b6acb4ec5\") " pod="openshift-must-gather-t4cqb/must-gather-cl655" Oct 30 13:10:00.404081 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.404014 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfmw8\" (UniqueName: \"kubernetes.io/projected/518c6d68-4e00-48ad-9e4f-5d39c3043e02-kube-api-access-lfmw8\") pod \"must-gather-vsvdr\" (UID: \"518c6d68-4e00-48ad-9e4f-5d39c3043e02\") " pod="openshift-must-gather-47mpn/must-gather-vsvdr" Oct 30 13:10:00.404081 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.404033 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/86f0d9f7-b36b-4129-8360-a6ed3dfcf610-must-gather-output\") pod \"must-gather-hfrq7\" (UID: \"86f0d9f7-b36b-4129-8360-a6ed3dfcf610\") " pod="openshift-must-gather-wls7l/must-gather-hfrq7" Oct 30 13:10:00.404210 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.404141 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/518c6d68-4e00-48ad-9e4f-5d39c3043e02-must-gather-output\") pod \"must-gather-vsvdr\" (UID: \"518c6d68-4e00-48ad-9e4f-5d39c3043e02\") " pod="openshift-must-gather-47mpn/must-gather-vsvdr" Oct 30 13:10:00.404265 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.404214 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjsrh\" (UniqueName: \"kubernetes.io/projected/86f0d9f7-b36b-4129-8360-a6ed3dfcf610-kube-api-access-zjsrh\") pod \"must-gather-hfrq7\" (UID: \"86f0d9f7-b36b-4129-8360-a6ed3dfcf610\") " pod="openshift-must-gather-wls7l/must-gather-hfrq7" Oct 30 13:10:00.404383 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.404362 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/86f0d9f7-b36b-4129-8360-a6ed3dfcf610-must-gather-output\") pod \"must-gather-hfrq7\" (UID: \"86f0d9f7-b36b-4129-8360-a6ed3dfcf610\") " pod="openshift-must-gather-wls7l/must-gather-hfrq7" Oct 30 13:10:00.404457 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.404362 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/958c13ff-fd33-4c6c-831d-483b6acb4ec5-must-gather-output\") pod \"must-gather-cl655\" (UID: \"958c13ff-fd33-4c6c-831d-483b6acb4ec5\") " pod="openshift-must-gather-t4cqb/must-gather-cl655" Oct 30 13:10:00.444103 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.444059 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82ql9\" (UniqueName: \"kubernetes.io/projected/958c13ff-fd33-4c6c-831d-483b6acb4ec5-kube-api-access-82ql9\") pod \"must-gather-cl655\" (UID: \"958c13ff-fd33-4c6c-831d-483b6acb4ec5\") " pod="openshift-must-gather-t4cqb/must-gather-cl655" Oct 30 13:10:00.452010 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.451987 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjsrh\" (UniqueName: \"kubernetes.io/projected/86f0d9f7-b36b-4129-8360-a6ed3dfcf610-kube-api-access-zjsrh\") pod \"must-gather-hfrq7\" (UID: \"86f0d9f7-b36b-4129-8360-a6ed3dfcf610\") " pod="openshift-must-gather-wls7l/must-gather-hfrq7" Oct 30 13:10:00.504846 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.504825 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/518c6d68-4e00-48ad-9e4f-5d39c3043e02-must-gather-output\") pod \"must-gather-vsvdr\" (UID: \"518c6d68-4e00-48ad-9e4f-5d39c3043e02\") " pod="openshift-must-gather-47mpn/must-gather-vsvdr" Oct 30 13:10:00.504978 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.504930 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfmw8\" (UniqueName: \"kubernetes.io/projected/518c6d68-4e00-48ad-9e4f-5d39c3043e02-kube-api-access-lfmw8\") pod \"must-gather-vsvdr\" (UID: \"518c6d68-4e00-48ad-9e4f-5d39c3043e02\") " pod="openshift-must-gather-47mpn/must-gather-vsvdr" Oct 30 13:10:00.505200 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.505176 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/518c6d68-4e00-48ad-9e4f-5d39c3043e02-must-gather-output\") pod \"must-gather-vsvdr\" (UID: \"518c6d68-4e00-48ad-9e4f-5d39c3043e02\") " pod="openshift-must-gather-47mpn/must-gather-vsvdr" Oct 30 13:10:00.544222 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.544176 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfmw8\" (UniqueName: \"kubernetes.io/projected/518c6d68-4e00-48ad-9e4f-5d39c3043e02-kube-api-access-lfmw8\") pod \"must-gather-vsvdr\" (UID: \"518c6d68-4e00-48ad-9e4f-5d39c3043e02\") " pod="openshift-must-gather-47mpn/must-gather-vsvdr" Oct 30 13:10:00.577517 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.577500 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wls7l/must-gather-hfrq7" Oct 30 13:10:00.583360 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.583344 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t4cqb/must-gather-cl655" Oct 30 13:10:00.671472 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.671447 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47mpn/must-gather-vsvdr" Oct 30 13:10:00.880016 ip-10-0-131-76 kubenswrapper[4468]: W1030 13:10:00.879979 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86f0d9f7_b36b_4129_8360_a6ed3dfcf610.slice/crio-2d039242f23cc87892e532d4c73971b04dc551eab396b0f2d3d217e613314af9 WatchSource:0}: Error finding container 2d039242f23cc87892e532d4c73971b04dc551eab396b0f2d3d217e613314af9: Status 404 returned error can't find the container with id 2d039242f23cc87892e532d4c73971b04dc551eab396b0f2d3d217e613314af9 Oct 30 13:10:00.881222 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.881202 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wls7l/must-gather-hfrq7"] Oct 30 13:10:00.881891 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.881875 4468 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:10:00.893986 ip-10-0-131-76 kubenswrapper[4468]: W1030 13:10:00.893961 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod958c13ff_fd33_4c6c_831d_483b6acb4ec5.slice/crio-985d622bb80aa19338424635a78d384ed81b20b4b71f9b8af5b759c183a3c8ca WatchSource:0}: Error finding container 985d622bb80aa19338424635a78d384ed81b20b4b71f9b8af5b759c183a3c8ca: Status 404 returned error can't find the container with id 985d622bb80aa19338424635a78d384ed81b20b4b71f9b8af5b759c183a3c8ca Oct 30 13:10:00.895935 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.895915 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t4cqb/must-gather-cl655"] Oct 30 13:10:00.979103 ip-10-0-131-76 kubenswrapper[4468]: W1030 13:10:00.979063 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod518c6d68_4e00_48ad_9e4f_5d39c3043e02.slice/crio-4c6651af82feba63d4da1bd44d9289cac839f78dbfa83fda5cc47194f9ba7382 WatchSource:0}: Error finding container 4c6651af82feba63d4da1bd44d9289cac839f78dbfa83fda5cc47194f9ba7382: Status 404 returned error can't find the container with id 4c6651af82feba63d4da1bd44d9289cac839f78dbfa83fda5cc47194f9ba7382 Oct 30 13:10:00.979582 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:00.979565 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47mpn/must-gather-vsvdr"] Oct 30 13:10:01.150023 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:01.149954 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wls7l/must-gather-hfrq7" event={"ID":"86f0d9f7-b36b-4129-8360-a6ed3dfcf610","Type":"ContainerStarted","Data":"2d039242f23cc87892e532d4c73971b04dc551eab396b0f2d3d217e613314af9"} Oct 30 13:10:01.151057 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:01.151035 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t4cqb/must-gather-cl655" event={"ID":"958c13ff-fd33-4c6c-831d-483b6acb4ec5","Type":"ContainerStarted","Data":"985d622bb80aa19338424635a78d384ed81b20b4b71f9b8af5b759c183a3c8ca"} Oct 30 13:10:01.152098 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:01.152065 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47mpn/must-gather-vsvdr" event={"ID":"518c6d68-4e00-48ad-9e4f-5d39c3043e02","Type":"ContainerStarted","Data":"4c6651af82feba63d4da1bd44d9289cac839f78dbfa83fda5cc47194f9ba7382"} Oct 30 13:10:02.160641 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:02.160609 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47mpn/must-gather-vsvdr" event={"ID":"518c6d68-4e00-48ad-9e4f-5d39c3043e02","Type":"ContainerStarted","Data":"0081c9ffb2011b71048ca06910f9dce22907f77777085a4d8944ce70ae0c5071"} Oct 30 13:10:03.174807 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:03.174765 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47mpn/must-gather-vsvdr" event={"ID":"518c6d68-4e00-48ad-9e4f-5d39c3043e02","Type":"ContainerStarted","Data":"4dc96b4b3f87d29d97b569d361c5a1c7b48f4cf321af516ffc8dae423d634778"} Oct 30 13:10:03.219037 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:03.218979 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-47mpn/must-gather-vsvdr" podStartSLOduration=2.398706915 podStartE2EDuration="3.218959919s" totalImagesPullingTime="820.253004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:03.218395251 +0000 UTC m=+4554.252533648" watchObservedRunningTime="2026-10-30 13:10:03.218959919 +0000 UTC m=+4554.253098318" Oct 30 13:10:06.705486 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:06.705448 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mqv9x_46289f03-406c-4831-9ad1-0e6751fd7bd5/global-pull-secret-syncer/0.log" Oct 30 13:10:07.713092 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:07.709702 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qjvts_f15ba445-e38a-46f6-be0f-65f7307e79d6/konnectivity-agent/0.log" Oct 30 13:10:08.029956 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:08.028912 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-76.ec2.internal_5a981c242c496637c2b85dbce301f463/haproxy/0.log" Oct 30 13:10:15.412088 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:15.410360 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wls7l/must-gather-hfrq7" event={"ID":"86f0d9f7-b36b-4129-8360-a6ed3dfcf610","Type":"ContainerStarted","Data":"2324af1edc998c0e1d6985e054e2a8820fe79bc5d080f85c940e0e503d4d4b80"} Oct 30 13:10:16.212012 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:16.211977 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5949f4fc4c-bpf6c_0d4dab63-54cf-49ea-98a6-cfbf3adc9b68/authorino/0.log" Oct 30 13:10:16.433850 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:16.433804 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wls7l/must-gather-hfrq7" event={"ID":"86f0d9f7-b36b-4129-8360-a6ed3dfcf610","Type":"ContainerStarted","Data":"e46b85ac3e637c64da9d3c71b6519e6f20df5d7a2e5265dd35c45261b80584a2"} Oct 30 13:10:16.484560 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:16.481365 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wls7l/must-gather-hfrq7" podStartSLOduration=2.325783224 podStartE2EDuration="16.48134657s" totalImagesPullingTime="14.155563346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:16.478594205 +0000 UTC m=+4567.512732603" watchObservedRunningTime="2026-10-30 13:10:16.48134657 +0000 UTC m=+4567.515484968" Oct 30 13:10:17.453089 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:17.452598 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t4cqb/must-gather-cl655" event={"ID":"958c13ff-fd33-4c6c-831d-483b6acb4ec5","Type":"ContainerStarted","Data":"726e51b62d2f09b6f63f2ec0a752a2723ba9d1cd029d1e16a83a5e2dca1b9b0b"} Oct 30 13:10:17.453089 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:17.452648 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t4cqb/must-gather-cl655" event={"ID":"958c13ff-fd33-4c6c-831d-483b6acb4ec5","Type":"ContainerStarted","Data":"7a959d201f86557a7ff5aacd48a809d3acb74a535cee9d4b282679c3f4339d07"} Oct 30 13:10:17.525091 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:17.521907 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-t4cqb/must-gather-cl655" podStartSLOduration=2.024491211 podStartE2EDuration="17.521886889s" totalImagesPullingTime="15.497395678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:17.501390314 +0000 UTC m=+4568.535528712" watchObservedRunningTime="2026-10-30 13:10:17.521886889 +0000 UTC m=+4568.556025287" Oct 30 13:10:18.354806 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.354769 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6858bfbcf8-h9qws_bc2e5679-e297-4aae-8676-b55f902809b1/manager/0.log" Oct 30 13:10:18.490158 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.489666 4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5"] Oct 30 13:10:18.500096 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.497754 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" Oct 30 13:10:18.527113 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.525876 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5"] Oct 30 13:10:18.633881 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.633790 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5643a520-8b35-416a-9401-ff10f75350ed-proc\") pod \"perf-node-gather-daemonset-t8vj5\" (UID: \"5643a520-8b35-416a-9401-ff10f75350ed\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" Oct 30 13:10:18.633881 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.633855 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcz4v\" (UniqueName: \"kubernetes.io/projected/5643a520-8b35-416a-9401-ff10f75350ed-kube-api-access-lcz4v\") pod \"perf-node-gather-daemonset-t8vj5\" (UID: \"5643a520-8b35-416a-9401-ff10f75350ed\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" Oct 30 13:10:18.634099 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.633964 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5643a520-8b35-416a-9401-ff10f75350ed-sys\") pod \"perf-node-gather-daemonset-t8vj5\" (UID: \"5643a520-8b35-416a-9401-ff10f75350ed\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" Oct 30 13:10:18.634099 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.634023 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5643a520-8b35-416a-9401-ff10f75350ed-podres\") pod \"perf-node-gather-daemonset-t8vj5\" (UID: \"5643a520-8b35-416a-9401-ff10f75350ed\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" Oct 30 13:10:18.634338 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.634304 4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5643a520-8b35-416a-9401-ff10f75350ed-lib-modules\") pod \"perf-node-gather-daemonset-t8vj5\" (UID: \"5643a520-8b35-416a-9401-ff10f75350ed\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" Oct 30 13:10:18.741107 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.738919 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5643a520-8b35-416a-9401-ff10f75350ed-proc\") pod \"perf-node-gather-daemonset-t8vj5\" (UID: \"5643a520-8b35-416a-9401-ff10f75350ed\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" Oct 30 13:10:18.741107 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.738980 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcz4v\" (UniqueName: \"kubernetes.io/projected/5643a520-8b35-416a-9401-ff10f75350ed-kube-api-access-lcz4v\") pod \"perf-node-gather-daemonset-t8vj5\" (UID: \"5643a520-8b35-416a-9401-ff10f75350ed\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" Oct 30 13:10:18.741107 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.739008 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5643a520-8b35-416a-9401-ff10f75350ed-sys\") pod \"perf-node-gather-daemonset-t8vj5\" (UID: \"5643a520-8b35-416a-9401-ff10f75350ed\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" Oct 30 13:10:18.741107 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.739048 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5643a520-8b35-416a-9401-ff10f75350ed-podres\") pod \"perf-node-gather-daemonset-t8vj5\" (UID: \"5643a520-8b35-416a-9401-ff10f75350ed\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" Oct 30 13:10:18.741107 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.739174 4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5643a520-8b35-416a-9401-ff10f75350ed-lib-modules\") pod \"perf-node-gather-daemonset-t8vj5\" (UID: \"5643a520-8b35-416a-9401-ff10f75350ed\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" Oct 30 13:10:18.741107 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.739327 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5643a520-8b35-416a-9401-ff10f75350ed-lib-modules\") pod \"perf-node-gather-daemonset-t8vj5\" (UID: \"5643a520-8b35-416a-9401-ff10f75350ed\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" Oct 30 13:10:18.741107 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.739383 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5643a520-8b35-416a-9401-ff10f75350ed-proc\") pod \"perf-node-gather-daemonset-t8vj5\" (UID: \"5643a520-8b35-416a-9401-ff10f75350ed\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" Oct 30 13:10:18.741107 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.739696 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5643a520-8b35-416a-9401-ff10f75350ed-sys\") pod \"perf-node-gather-daemonset-t8vj5\" (UID: \"5643a520-8b35-416a-9401-ff10f75350ed\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" Oct 30 13:10:18.741107 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.739753 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5643a520-8b35-416a-9401-ff10f75350ed-podres\") pod \"perf-node-gather-daemonset-t8vj5\" (UID: \"5643a520-8b35-416a-9401-ff10f75350ed\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" Oct 30 13:10:18.769095 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.768316 4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcz4v\" (UniqueName: \"kubernetes.io/projected/5643a520-8b35-416a-9401-ff10f75350ed-kube-api-access-lcz4v\") pod \"perf-node-gather-daemonset-t8vj5\" (UID: \"5643a520-8b35-416a-9401-ff10f75350ed\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" Oct 30 13:10:18.829417 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:18.829378 4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" Oct 30 13:10:19.175955 ip-10-0-131-76 kubenswrapper[4468]: W1030 13:10:19.175602 4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5643a520_8b35_416a_9401_ff10f75350ed.slice/crio-bdf246d41f40db67f862a170e158357465246fc19de1400af82b5b3a229fe9df WatchSource:0}: Error finding container bdf246d41f40db67f862a170e158357465246fc19de1400af82b5b3a229fe9df: Status 404 returned error can't find the container with id bdf246d41f40db67f862a170e158357465246fc19de1400af82b5b3a229fe9df Oct 30 13:10:19.179929 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:19.179903 4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5"] Oct 30 13:10:19.292542 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:19.292504 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb8f9449b-nnd2b_731fb32a-31aa-4d57-8edd-4f890b9e9288/manager/0.log" Oct 30 13:10:19.467138 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:19.466659 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb8f9449b-nnd2b_731fb32a-31aa-4d57-8edd-4f890b9e9288/manager/1.log" Oct 30 13:10:19.470125 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:19.470064 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" event={"ID":"5643a520-8b35-416a-9401-ff10f75350ed","Type":"ContainerStarted","Data":"18959f1251b6bef9ff8c4445401741d9fc9ead66176618429be2e94b1dc4e03a"} Oct 30 13:10:19.470686 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:19.470665 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" Oct 30 13:10:19.470792 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:19.470703 4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" event={"ID":"5643a520-8b35-416a-9401-ff10f75350ed","Type":"ContainerStarted","Data":"bdf246d41f40db67f862a170e158357465246fc19de1400af82b5b3a229fe9df"} Oct 30 13:10:19.522593 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:19.522532 4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" podStartSLOduration=1.522513514 podStartE2EDuration="1.522513514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:19.520428688 +0000 UTC m=+4570.554567084" watchObservedRunningTime="2026-10-30 13:10:19.522513514 +0000 UTC m=+4570.556651911" Oct 30 13:10:19.662937 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:19.662869 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6858bfbcf8-h9qws_bc2e5679-e297-4aae-8676-b55f902809b1/manager/0.log" Oct 30 13:10:20.704279 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:20.704101 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb8f9449b-nnd2b_731fb32a-31aa-4d57-8edd-4f890b9e9288/manager/0.log" Oct 30 13:10:20.898855 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:20.898794 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb8f9449b-nnd2b_731fb32a-31aa-4d57-8edd-4f890b9e9288/manager/1.log" Oct 30 13:10:21.188319 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:21.188289 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-j7nfs_fb5b969c-76f1-47a1-8fc0-00fac17f17d0/proxy/0.log" Oct 30 13:10:22.292273 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:22.288383 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m2dqw_bf169833-c766-4b19-a563-092d490f0090/node-exporter/0.log" Oct 30 13:10:22.410335 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:22.410303 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m2dqw_bf169833-c766-4b19-a563-092d490f0090/kube-rbac-proxy/0.log" Oct 30 13:10:22.438182 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:22.438151 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-j7nfs_fb5b969c-76f1-47a1-8fc0-00fac17f17d0/proxy/0.log" Oct 30 13:10:22.539123 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:22.539054 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m2dqw_bf169833-c766-4b19-a563-092d490f0090/init-textfile/0.log" Oct 30 13:10:22.685846 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:22.685816 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wrq8c_aa996c80-1b20-4275-b2ce-26bb5b7d08e0/kube-rbac-proxy-main/0.log" Oct 30 13:10:22.807326 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:22.807292 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wrq8c_aa996c80-1b20-4275-b2ce-26bb5b7d08e0/kube-rbac-proxy-self/0.log" Oct 30 13:10:22.833052 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:22.833014 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6858bfbcf8-h9qws_bc2e5679-e297-4aae-8676-b55f902809b1/manager/0.log" Oct 30 13:10:22.929627 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:22.929600 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wrq8c_aa996c80-1b20-4275-b2ce-26bb5b7d08e0/openshift-state-metrics/0.log" Oct 30 13:10:23.604976 ip-10-0-131-76 kubenswrapper[4468]: E1030 13:10:23.604933 4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6dvql" containerName="extractor" Oct 30 13:10:23.793632 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:23.793592 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb8f9449b-nnd2b_731fb32a-31aa-4d57-8edd-4f890b9e9288/manager/0.log" Oct 30 13:10:23.973779 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:23.973740 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb8f9449b-nnd2b_731fb32a-31aa-4d57-8edd-4f890b9e9288/manager/1.log" Oct 30 13:10:24.097375 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:24.097346 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-h8pv8_ffa08fe5-b4b9-4775-9d46-486c0065135a/prometheus-operator/0.log" Oct 30 13:10:24.167841 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:24.167807 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6858bfbcf8-h9qws_bc2e5679-e297-4aae-8676-b55f902809b1/manager/0.log" Oct 30 13:10:24.230926 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:24.230861 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-h8pv8_ffa08fe5-b4b9-4775-9d46-486c0065135a/kube-rbac-proxy/0.log" Oct 30 13:10:24.371092 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:24.367907 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-7wdvn_f6144c8e-3940-4027-a3b2-04792894d8bc/prometheus-operator-admission-webhook/0.log" Oct 30 13:10:25.221921 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:25.221138 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb8f9449b-nnd2b_731fb32a-31aa-4d57-8edd-4f890b9e9288/manager/0.log" Oct 30 13:10:25.422091 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:25.421731 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb8f9449b-nnd2b_731fb32a-31aa-4d57-8edd-4f890b9e9288/manager/1.log" Oct 30 13:10:25.488226 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:25.488154 4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-t8vj5" Oct 30 13:10:25.568454 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:25.568427 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-j7nfs_fb5b969c-76f1-47a1-8fc0-00fac17f17d0/proxy/0.log" Oct 30 13:10:26.935427 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:26.935376 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-j7nfs_fb5b969c-76f1-47a1-8fc0-00fac17f17d0/proxy/0.log" Oct 30 13:10:27.442318 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:27.442284 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6858bfbcf8-h9qws_bc2e5679-e297-4aae-8676-b55f902809b1/manager/0.log" Oct 30 13:10:28.729330 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:28.726382 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb8f9449b-nnd2b_731fb32a-31aa-4d57-8edd-4f890b9e9288/manager/0.log" Oct 30 13:10:28.836366 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:28.836323 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6858bfbcf8-h9qws_bc2e5679-e297-4aae-8676-b55f902809b1/manager/0.log" Oct 30 13:10:28.967358 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:28.967319 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb8f9449b-nnd2b_731fb32a-31aa-4d57-8edd-4f890b9e9288/manager/1.log" Oct 30 13:10:30.080089 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:30.076450 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb8f9449b-nnd2b_731fb32a-31aa-4d57-8edd-4f890b9e9288/manager/0.log" Oct 30 13:10:30.213275 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:30.213241 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-77449bbc64-2hhck_6b6c009e-76fe-4449-88d2-4f4d65d44b7b/console/0.log" Oct 30 13:10:30.280649 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:30.280621 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb8f9449b-nnd2b_731fb32a-31aa-4d57-8edd-4f890b9e9288/manager/1.log" Oct 30 13:10:30.356696 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:30.356550 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-4mb5r_24aac159-d3bb-4ed2-8132-93456b41cefe/download-server/0.log" Oct 30 13:10:30.939950 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:30.939917 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-j7nfs_fb5b969c-76f1-47a1-8fc0-00fac17f17d0/proxy/0.log" Oct 30 13:10:31.931735 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:31.931161 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-j7nfs_fb5b969c-76f1-47a1-8fc0-00fac17f17d0/proxy/0.log" Oct 30 13:10:34.549620 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:34.549588 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p4nvb_1a948657-c616-435c-90db-b880a0db5bd9/dns/0.log" Oct 30 13:10:34.669106 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:34.669040 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p4nvb_1a948657-c616-435c-90db-b880a0db5bd9/kube-rbac-proxy/0.log" Oct 30 13:10:35.161421 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:35.161318 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nhbdl_617427c7-590d-4d80-8c4e-e11c7812dca0/dns-node-resolver/0.log" Oct 30 13:10:36.546723 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:36.546685 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8447c7c9c8-ptjpp_9884f855-dc2f-4dc0-abbe-7ddf956f4cd5/registry/0.log" Oct 30 13:10:36.717489 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:36.717450 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7c297_29d0c5b3-0ef9-4e82-bd58-3fe952dd0d9e/node-ca/0.log" Oct 30 13:10:37.093919 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:37.093878 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68fcc45f6c-fbxv7_b4d73c96-8ad8-4aba-908c-88f4969524c1/kube-auth-proxy/0.log" Oct 30 13:10:37.249972 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:37.249942 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-7tbjq_d6305f49-03f1-4d47-a0a7-19776c672543/istio-proxy/0.log" Oct 30 13:10:37.459743 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:37.459713 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-94zmh_a1d28f84-07c8-4c39-830b-608b124282f5/payload-pre-processing/0.log" Oct 30 13:10:37.616469 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:37.616429 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-ftmv4_910664f0-5127-40e1-b2c8-493f050457cc/payload-processing/0.log" Oct 30 13:10:38.184021 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:38.183953 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68fcc45f6c-fbxv7_b4d73c96-8ad8-4aba-908c-88f4969524c1/kube-auth-proxy/0.log" Oct 30 13:10:38.269501 ip-10-0-131-76 kubenswrapper[4468]: I1030 13:10:38.269471 4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68fcc45f6c-fbxv7_b4d73c96-8ad8-4aba-908c-88f4969524c1/kube-auth-proxy/0.log"